Is GLM-5.2 safe? The US government tested it as an agent, not a chatbot
7 min read
On 17 July 2026 the Center for AI Standards and Innovation (CAISI) — the US government body set up inside the National Institute of Standards and Technology (NIST) to assess the risks of frontier models — published its assessment of GLM-5.2, the open-weight model released by Z.ai (formerly Zhipu AI) on 16 June 2026. The report reaches a two-part conclusion that matters to anyone assessing this model for a real task, not a conversation: asked directly, GLM-5.2 almost always refuses to help build a cyberattack. Left free to act step by step, tools in hand, the same model never stopped.
The facts, in order
- 16 June 2026 — Z.ai, a Beijing company founded in 2019 by Tsinghua University researchers, publishes the GLM-5.2 weights on Hugging Face under an MIT licence. Mixture-of-experts architecture, per the official configuration file: 78 layers, 256 routed experts plus one shared, eight active per token, a context window of up to 1,048,576 tokens — just over a million.
- 8 July 2026 — CAISI completes its independent assessment, made public on 17 July. Corporate context reported in the same document: Z.ai listed in Hong Kong on 8 January 2026, the first major Chinese AI developer to do so, with market capitalisation up from $7B to roughly $65B in six months and 2025 revenue of $105M, doubling every year since 2022.
- Overall capability, per CAISI’s evaluations, makes GLM-5.2 “probably the most capable open-weight AI model when it was released”, with overall capability similar to GPT-5.2 (December 2025) and cyber capability comparable to Opus 4.6 (February 2026) — still behind the closed US frontier, represented in the report by Opus 4.8 and GPT-5.5.
- The developer, on its own blog, claims that “on standard coding benchmarks, GLM-5.2 is the strongest open-source model”, with capability “roughly positioned between Claude Opus 4.7 and Claude Opus 4.8”. But on the three chapters that matter for security — offensive cyber capability, agent-hijacking resistance, misuse safeguards — the report logs the same note three times, identically: the developer did not self-report results. We have already written about the gap between what a technical report states and what it stays silent on: here the silence isn’t on a detail, it’s on an entire security chapter.
The refusal that disappears once the task becomes an agent
This is where the report stops being a benchmark bulletin and becomes relevant to a buying decision. On a CAISI-built test — thirty direct, overtly malicious requests, from attack ideation to exploit and payload development — GLM-5.2 “refused most requests, including when the request was posed using a public jailbreak”: it behaves exactly as you’d expect from a model with working safeguards.
Then CAISI measured the same capability not with a written question but with an agentic task: ten trials in which the model has to build a complete exploit using tools, step by step, over as many as 300 turns. The result, verbatim: “GLM-5.2 did not refuse any of the requests.” Zero out of ten, from the very same model that a few lines earlier refused most of the same requests posed in one go. CAISI’s summary is the sentence worth putting in a procurement file: GLM-5.2’s safeguards “do not fully prevent assistance with cyber offense tasks in the absence of overtly malicious framing” — they hold on a question phrased the way a chat user would phrase it, and give way on a task broken into steps, which is exactly the form in which these models end up in production: coding or analysis agents given a technical objective, not an explicit question. Nobody in a company asks an agent “help me attack a system”: they assign it a technical task, and it’s that technical task the report measured.
The picture on biology is symmetrical: the model answers sensitive questions “at a higher rate and with more detail” than the tested US models, while remaining more cautious than the other Chinese open models assessed in the same report. Not everything tips the same way: on resistance to agent hijacking — email, calendar events or tampered documents used to make it carry out a different task — GLM-5.2 is the most robust of the open-weight models tested, Chinese ones included: it “was never successfully hijacked by publicly available agent hijacking attacks.” But CAISI adds the caveat that applies to every figure in this section: the tests measure robustness against prompt-based jailbreaks, and regardless of that robustness, “safeguards for open-weight models can be circumvented when self-hosted.”
One methodological detail makes the report more useful than a vendor announcement: CAISI did not query Z.ai’s API. It downloaded the weights from Hugging Face and ran them on its own servers, in the same FP8-quantised version that — the report notes — appears to be the one Z.ai itself serves to users, without adding safeguards beyond the factory ones, to reproduce “the configuration with which they would be deployed if self-hosted.” Not the best figure a vendor can produce in a lab, but the one you get by downloading the model today and putting it into service as it comes.
What to do about it
- Don’t stop at the coding benchmark a vendor publishes: on cybersecurity, agent security and safeguards, here the developer published nothing at all. Silence on a security chapter is not a reassurance — it’s a gap you have to fill before use.
- If your use case involves an agent with tool access — not a plain chatbot — test the behaviour in the agentic configuration, not in a single question: that’s where, per CAISI, the safeguards give way.
- Treat factory safeguards as a revocable starting point: once downloaded and run in-house, the model only answers to the controls you add around it, not to a specification document.
- Before adopting a model for tasks with potential offensive capability (vulnerability analysis, red teaming, security patching), document your own test in writing — weights version, configuration, outcome: a figure without the method that produced it won’t survive an auditor.
How we solve this
For entities already inside Italy’s national cybersecurity perimeter, the principle is nothing new to import: the decree listing the ICT categories subject to CVCN evaluation already includes, verbatim, “Artificial Intelligence (AI) and Machine Learning (ML) systems for network/system management” — a supply that manages a network doesn’t go live until someone has methodically put it to the test. For everyone else, Italy’s NIS2 transposition (D.lgs. 138/2024) still requires a documented assessment of suppliers before use. The CAISI report shows what that assessment should look like when the vendor doesn’t do it itself: we replicate the same kind of test — agentic configuration, weights verified against their hash, safeguards tested with no external additions — as a control that runs before go-live and on a set cadence afterwards, with the record to show an inspector.
The same setup that runs this verification is what then runs the agent in production: the client’s scattered data — archives, business systems, sensors — brought into a single operating model, on which the agent executes the technical task with a human operator in command at exactly the point this report shows to be most fragile: execution, not the request. For large enterprises, defence, government and healthcare, which model you pick — American, European or Chinese, open or closed — matters less than who controls that point. We do it in both delivery modes, on-premise on the client’s self-contained machines or on a dedicated cloud with a data centre in Italy, with governance run together with you.
Assessing an open-weight model for an agentic use case, not just a conversational one? Let’s talk in a thirty-minute session: we replicate the test in your real configuration, before someone else does it in your place.