IT EN
Start a trial

Operational notes Observatory

276 billion parameters, 12 active: what it actually takes to run in-house

7 min read

Close-up of a wax honeycomb: thousands of empty hexagonal cells, in black and white
Two hundred and fifty-six cells built, six walked through on each pass: you pay for all the memory, not for all the compute.

Today, 31 July 2026, Thinking Machines Lab published the weights of Inkling-Small, a multimodal model under the Apache 2.0 licence. The product page introduces it with a line that is already a sales argument: it matches its larger sibling on many benchmarks at a quarter of the size, with lower cost and latency. The two headline numbers are 276 billion total parameters and 12 billion active. They do not measure the same thing and they do not land in the same budget line: one tells you how much memory to buy, the other what each token costs. Confusing them is the fastest way to get a six-figure sizing exercise wrong.

What is declared

  • Architecture: a 42-layer decoder-only transformer with a sparse Mixture-of-Experts backbone. Each token is routed to 6 experts out of 256, plus 2 shared experts active on every token. Attention is a hybrid of local and global layers.
  • Inputs: text, images (each dimension ideally between 40 and 4,096 pixels) and audio (WAV at 16 kHz, ideally under two minutes). Output: text only.
  • Numerics: BF16 and NVFP4, in two separate repositories. Local deployment: SGLang, vLLM, TokenSpeed, Unsloth and Hugging Face Transformers, each with its own recipe linked from the card.
  • Context: the model card does not state a context window. The one-million-token figure sits on the product page, which in the same panel qualifies it as 64K / 256K on Tinker, the vendor’s own platform. Those are two different statements.

Total and active: two numbers, two cost lines

Active parameters govern the compute per token. That is where speed and energy per response come from, and it is why a sparse model with 12 billion active parameters answers like a dense model of that size. Total parameters govern memory: all of them have to sit somewhere, and all of them have to be reachable quickly. Faced with “12 billion active”, the temptation is to size for twelve. It does not work, and the reason is in the architecture quoted above: routing changes on every token. The six experts needed for the next token are not the six in use now, and keeping only the active slice resident means fetching the missing weights from somewhere else at every step.

Bytes, unlike round numbers, can be measured. The public Hugging Face API reports that the BF16 repository occupies 531,944,513,947 bytes, about 531.9 GB, and the NVFP4 variant 170,764,725,303 bytes, about 170.8 GB — a factor of 3.1. The breakdown of the compressed build is instructive: 251,255,586,816 four-bit values, plus 15,703,474,176 FP8 scale factors, exactly one per sixteen values, as NVIDIA’s NVFP4 format prescribes and as NVIDIA documents at 4.5 bits per value, executed natively by accelerators from the Blackwell architecture onwards. A further 14,700,842,072 parameters remain in BF16: quantisation does not touch everything. Averaged across the file, a little over five bits per parameter. That is 170.8 GB of weights alone, before the context cache, the activations and whatever the inference engine adds. We are not redoing the hardware arithmetic here: it lives on the page on hardware and open-weight models.

One thing does not reconcile, and we will say so. The tensor index counts 265,956,439,090 parameters, close to 266 billion rather than 276; the BF16 file size confirms it, at exactly two bytes per parameter. The gap is roughly 4% and we have no verified explanation for it: it may be how the vendor counts, unpublished tensors, or the multimodal encoders. If you need that figure for sizing, use the bytes.

What shrinking costs you

The evaluation table the vendor published is more candid than the announcement, and it should be read down the column. On SWEBench Verified, Inkling-Small reports 80.2% against 77.6% for the 975-billion sibling: the small one is ahead. On GPQA Diamond, 89.5% against 87.2%; on audio and vision the two are level. Then you reach factuality and the sign flips: SimpleQA Verified 20.6% against 43.9%, AA Omniscience index −9.0 against +2.1. That is the price of total parameters. Memorised knowledge lives there, and removing two-thirds of them removes some of it.

It is the row that matters most in an adoption decision, because it separates two use cases. If the system has to answer over your documents — retrieval, extraction, comparison against requirements — the model’s encyclopaedic memory counts for little: you supply the facts. If somebody instead plans to use it as a general oracle, that 20.6% measures how often it will invent one confidently. A third reading, on domain agentic work: on Tau 3 Banking the score is 15.5%, and the best model in the same table stops at 24.3%. Nobody should be automating a banking process on those numbers, with any of those models.

How to read that table

The footnotes below the table are worth as much as the table. The vendor states that the SWEBench Verified scores for its own two models come from an internal bash-only harness, while the figures for external models are self-reported; on Terminal Bench it uses another internal harness and notes that some solutions found to be contaminated by web search were scored zero. These are correct and unusually candid disclosures, but it is not the same test administered to everyone. What holds for what a technical report declares and what it leaves out holds here: the only measurements you can decide on are the ones you rerun yourself, on your own data.

An open licence, and a separate use policy

The licence is Apache 2.0, which closes the questions a custom licence such as Kimi K3’s leaves open: no revenue thresholds, no separate agreement. But the card also points to a Model Acceptable Use Policy, last updated 15 July 2026, which declares itself binding on anyone “accessing, downloading, or using” the materials and extends liability downstream: anyone who lets others use the model through their own products is responsible for their compliance too. The document does not mention Apache 2.0 and does not explain how the two sit together. For a procurement team, that is a line to put in front of legal before installation.

What to do, in practice

  • Size on the memory of the total parameters: the figure for the tender is the byte count, 170.8 GB for the compressed build and 531.9 GB for BF16, both before the context cache.
  • Have the contract state which numeric format the model will run in and on which accelerators: NVFP4 is native on Blackwell, where it is not you fall back to BF16 and memory triples.
  • Do not use the million-token window for sizing until you have verified it on your own engine: the product page itself quotes far shorter windows.
  • Decide up front whether the use case lives on your documents or on the model’s own knowledge. In the second case, the factuality gap is the main risk item.
  • Rerun the tests you care about with your own evaluation set: the numbers in that table are not all produced the same way, and the vendor says so.

How we handle it

A model that fits in a few hundred gigabytes can be installed almost anywhere, and that is precisely the problem: ease of installation says nothing about who sees the data while it runs. That is why we deliver in two modes and no others: on-premise in the client’s own environment, or on our dedicated cloud — reserved for the single client, access over a dedicated VPN, data centre resident in Italy, premises we staff ourselves. In both, the model is a replaceable component and not a permanent commitment, as it was for the earlier releases we assessed: it is the installation you build first, before choosing which weights run inside it.

Do you need to estimate what it takes to keep a sparse model of a few hundred billion parameters in-house? Half an hour with one of our specialists: we start from the measured bytes and your real volume, not from the figures in the announcement.

Sources