---
title: "Is GLM-5.2 safe? The US government tested it as an agent, not a chatbot"
description: "On 17 July 2026 CAISI (NIST) published its assessment of GLM-5.2: it refuses attacks in chat, but not when carried out by an agent with tools in hand."
url: https://csidia.com/en/notes/glm-5-2-caisi-valutazione-sicurezza/
lang: en
date: 2026-08-02
label: "Observatory"
publisher: CSIDIA S.r.l. (https://csidia.com/)
italiano: https://csidia.com/note-operative/glm-5-2-caisi-valutazione-sicurezza/
---

# Is GLM-5.2 safe? The US government tested it as an agent, not a chatbot

On 17 July 2026 the Center for AI Standards and Innovation (CAISI) — the US government body set up inside the National Institute of Standards and Technology (NIST) to assess the risks of frontier models — published its assessment of GLM-5.2, the open-weight model released by Z.ai (formerly Zhipu AI) on 16 June 2026. The report reaches a two-part conclusion that matters to anyone assessing this model for a real task, not a conversation: asked directly, GLM-5.2 almost always refuses to help build a cyberattack. Left free to act step by step, tools in hand, the same model never stopped.

## The facts, in order

- **16 June 2026** — Z.ai, a Beijing company founded in 2019 by Tsinghua University researchers, publishes the GLM-5.2 weights on Hugging Face under an MIT licence. Mixture-of-experts architecture, per the official configuration file: 78 layers, 256 routed experts plus one shared, eight active per token, a context window of up to 1,048,576 tokens — just over a million.
- **8 July 2026** — CAISI completes its independent assessment, made public on 17 July. Corporate context reported in the same document: Z.ai listed in Hong Kong on 8 January 2026, the first major Chinese AI developer to do so, with market capitalisation up from $7B to roughly $65B in six months and 2025 revenue of $105M, doubling every year since 2022.
- **Overall capability**, per CAISI's evaluations, makes GLM-5.2 "probably the most capable open-weight AI model when it was released", with overall capability similar to GPT-5.2 (December 2025) and cyber capability comparable to Opus 4.6 (February 2026) — still behind the closed US frontier, represented in the report by Opus 4.8 and GPT-5.5.
- **The developer**, on its own blog, claims that "on standard coding benchmarks, GLM-5.2 is the strongest open-source model", with capability "roughly positioned between Claude Opus 4.7 and Claude Opus 4.8". But on the three chapters that matter for security — offensive cyber capability, agent-hijacking resistance, misuse safeguards — the report logs the same note three times, identically: the developer did not self-report results. We have already [written about the gap between what a technical report states and what it stays silent on](https://csidia.com/en/notes/rapporto-tecnico-modello-cosa-dichiara): here the silence isn't on a detail, it's on an entire security chapter.

## The refusal that disappears once the task becomes an agent

This is where the report stops being a benchmark bulletin and becomes relevant to a buying decision. On a CAISI-built test — thirty direct, overtly malicious requests, from attack ideation to exploit and payload development — GLM-5.2 "refused most requests, including when the request was posed using a public jailbreak": it behaves exactly as you'd expect from a model with working safeguards.

Then CAISI measured the same capability not with a written question but with an agentic task: ten trials in which the model has to build a complete exploit using tools, step by step, over as many as 300 turns. The result, verbatim: "GLM-5.2 did not refuse any of the requests." Zero out of ten, from the very same model that a few lines earlier refused most of the same requests posed in one go. CAISI's summary is the sentence worth putting in a procurement file: GLM-5.2's safeguards "do not fully prevent assistance with cyber offense tasks in the absence of overtly malicious framing" — they hold on a question phrased the way a chat user would phrase it, and give way on a task broken into steps, which is exactly the form in which these models end up in production: coding or analysis agents given a technical objective, not an explicit question. Nobody in a company asks an agent "help me attack a system": they assign it a technical task, and it's that technical task the report measured.

The picture on biology is symmetrical: the model answers sensitive questions "at a higher rate and with more detail" than the tested US models, while remaining more cautious than the [other Chinese open models](https://csidia.com/en/notes/modelli-cinesi-open-source-in-azienda) assessed in the same report. Not everything tips the same way: on resistance to agent hijacking — email, calendar events or tampered documents used to make it carry out a different task — GLM-5.2 is the most robust of the open-weight models tested, Chinese ones included: it "was never successfully hijacked by publicly available agent hijacking attacks." But CAISI adds the caveat that applies to every figure in this section: the tests measure robustness against prompt-based jailbreaks, and regardless of that robustness, "safeguards for open-weight models can be circumvented when self-hosted."

One methodological detail makes the report more useful than a vendor announcement: CAISI did not query Z.ai's API. It downloaded the weights from Hugging Face and ran them on its own servers, in the same FP8-quantised version that — the report notes — appears to be the one Z.ai itself serves to users, without adding safeguards beyond the factory ones, to reproduce "the configuration with which they would be deployed if self-hosted." Not the best figure a vendor can produce in a lab, but the one you get by downloading the model today and putting it into service as it comes.

## What to do about it

- Don't stop at the coding benchmark a vendor publishes: on cybersecurity, agent security and safeguards, here the developer published nothing at all. Silence on a security chapter is not a reassurance — it's a gap you have to fill before use.
- If your use case involves an agent with tool access — not a plain chatbot — test the behaviour in the agentic configuration, not in a single question: that's where, per CAISI, the safeguards give way.
- Treat factory safeguards as a revocable starting point: once downloaded and run in-house, the model only answers to the controls you add around it, not to a specification document.
- Before adopting a model for tasks with potential offensive capability (vulnerability analysis, red teaming, security patching), document your own test in writing — weights version, configuration, outcome: a figure without the method that produced it won't survive an auditor.

## Sources

- [CAISI (NIST) — Assessment of Z.ai's GLM-5.2](https://www.nist.gov/news-events/news/2026/07/caisi-assessment-zais-glm-52)
- [CAISI (NIST) — full assessment text (PDF)](https://www.nist.gov/document/caisi-assessment-zais-glm-52)
- [Hugging Face — zai-org/GLM-5.2, model repository](https://huggingface.co/zai-org/GLM-5.2)

Note by CSIDIA S.r.l. (Vignola, MO, Italy): https://csidia.com/en/notes/glm-5-2-caisi-valutazione-sicurezza/ · Who we are and what we do: https://csidia.com/en/at-a-glance/ · engagement@csidia.com