Knowledge distillation was invented to squeeze big neural networks onto small devices. A decade later, it is compressing something else entirely: the gap between nations racing for AI supremacy.
In July 2026, a single post on X wiped out the comfortable assumption that artificial intelligence competition would be settled purely by who owned the most GPUs. Michael Kratsios, director of the White House Office of Science and Technology Policy, publicly accused China's Moonshot AI of building its remarkable Kimi K3 model by distilling Anthropic's frontier Fable model — alleging the company had built an internal platform capable of switching between access methods to evade detection. Within hours, the US Treasury Secretary was threatening sanctions and Entity List designations.
The technique at the center of this diplomatic firestorm is not espionage in the traditional sense. Nobody broke into a data center. Nobody stole a hard drive of model weights. The alleged act was, in essence, asking a model millions of questions and learning from its answers.
Welcome to the strangest intellectual property dispute of the decade — one where the "theft" looks almost identical to legitimate use, where the same technique is simultaneously celebrated as the backbone of efficient AI deployment and condemned as industrial-scale piracy, and where the line between innovation and imitation has never been blurrier.
This article unpacks what model distillation actually is, how it works under the hood, why every serious AI lab on Earth uses it, and how it became the newest front in the US–China technology conflict.
Part 1: What Is Model Distillation?
The teacher and the student
The largest AI systems — so-called frontier models — require staggering amounts of compute, data, and capital to train. They also require serious hardware just to run. A model with trillions of parameters cannot live on your phone, your car, or even most corporate servers.
Model distillation solves this with an elegantly simple idea: use a large, capable teacher model to train a smaller, cheaper student model. The teacher generates outputs — answers, reasoning chains, code, probability distributions — and the student learns to imitate them. The student does not inherit the teacher's weights or architecture. It learns selected behaviors, absorbing enough of the teacher's capability to perform specific tasks at a fraction of the cost.
The concept traces back to a 2015 paper by Geoffrey Hinton, Oriol Vinyals, and Jeff Dean at Google — "Distilling the Knowledge in a Neural Network." Their key insight was that a trained model's soft outputs contain far richer information than the hard labels it was trained on. When a teacher classifies an image as "dog," it also implicitly reveals that the image looks a little like "wolf," slightly like "cat," and nothing like "airplane." Hinton called this hidden relational information dark knowledge, and it turns out to be an extraordinarily effective training signal.
Interestingly, Jeff Dean — now leading AI at Google — recently recalled on a podcast that his team stumbled into distillation while developing Google's own models, long before the technique became a household term in tech policy circles.
Why everyone uses it
The economics are irresistible. Industry analyses in 2026 put the typical result of a well-executed distillation pipeline at:
- 5–30x reduction in inference cost
- Roughly 4x faster response times
- 95–97% of the teacher's performance retained on target tasks
DeepSeek provided the most famous demonstration. Its R1 reasoning model generated hundreds of thousands of high-quality chain-of-thought samples, which were then used to fine-tune smaller student models. One distilled student scored 94.5 on the MATH-500 benchmark — competitive with models many times its size. The International AI Safety Report 2026 noted that one such distillation-based fine-tune reportedly cost in the neighborhood of $10,000 — orders of magnitude below the cost of training comparable capability from scratch.
Distillation has quietly become mandatory infrastructure across the industry:
- Edge and mobile AI — assistants that run on-device rather than in the cloud
- Image and video generation — nearly every production diffusion system now includes an explicit distillation stage to hit sub-second generation targets
- Enterprise deployment — a distilled model specialized for your domain often beats renting a frontier model API, both on latency and cost
- Speculative decoding — small distilled "draft" models propose tokens that a large model verifies, dramatically accelerating inference
In short: distillation is not a fringe trick. It is how modern AI becomes affordable.
Part 2: How It Actually Works — For Architects and Data Scientists
If you build systems for a living, here is the machinery beneath the metaphor.
Soft labels and temperature
Classic (white-box) distillation trains the student to match the teacher's full output distribution, not just its top answer. Given logits z, the softmax is computed with a temperature parameter T:
p_i = exp(z_i / T) / Σ_j exp(z_j / T)
At T = 1 you get the normal distribution; raising T (say, 2–5) softens it, amplifying the small probabilities assigned to "wrong" classes — precisely the dark knowledge you want the student to absorb. The training objective is typically a weighted blend:
L = α · CE(y_true, student) + (1 − α) · T² · KL(teacher_T ‖ student_T)
where the KL-divergence term pulls the student's softened distribution toward the teacher's, and the T² factor keeps gradient magnitudes comparable across temperatures.
Black-box vs white-box
The crucial distinction for the geopolitics story:
- White-box distillation requires access to the teacher's logits or internal states. Only possible if you own the model or its weights are open.
- Black-box (sequence-level) distillation needs only the teacher's generated text. You prompt the model — through an API, like any customer — collect its outputs, and fine-tune your student on them as ordinary supervised data. DeepSeek's R1-Distill models are the textbook example of this hard-label approach.
Black-box distillation is why API access alone is enough to transfer capability — and why frontier labs now treat their inference endpoints as a security surface, not just a product.
Off-policy, on-policy, and self-distillation
By 2026, distillation inside frontier labs has evolved well beyond the original recipe. Practitioners now distinguish three regimes:
Off-policy distillation — the student trains on data the teacher generated (soft-label logit matching or hard-label text imitation). Simple, scalable, but the student never learns from its own mistakes, and exposure bias can accumulate over long generations.
On-policy distillation — the student generates its own rollouts, and the teacher grades every token, typically via a reverse KL objective (
KL(student ‖ teacher)). Reverse KL is mode-seeking: rather than smearing probability over everything the teacher might say, the student concentrates on doing its own generations the way the teacher would. This has become the dominant pattern for a very modern problem: labs train separate reinforcement-learning experts per domain — one for math, one for code, one for agentic tasks — because RL skills learned in one stage tend to degrade in the next. They then distill all the specialists into a single unified student while it generates on-policy. DeepSeek-V4's published pipeline describes exactly this architecture.Self-distillation — the model learns from a better version of itself (for example, its own best-of-N or extended-reasoning outputs), bootstrapping capability without any external teacher.
Practical considerations
A few hard-won lessons from the literature and production reports:
- Capacity gap: if the student is too small relative to the teacher, imitation fails. Intermediate "teaching assistant" models of medium size can bridge the gap.
- Data quality beats quantity: some 2026 research fine-tuned highly capable models with as few as ~1,000 carefully curated teacher samples. Curation of reasoning traces matters more than raw volume.
- Stacking compressions: distillation combines well with quantization and pruning; the compounding gains are how sub-second, on-device generation targets are being met.
- Multi-teacher conflicts: distilling from several teachers can produce contradictory gradients; sample routing and per-domain weighting are active research areas.
Part 3: The Geopolitical Firestorm
Act I — DeepSeek shakes the market (early 2025)
The political story begins in January 2025, when DeepSeek's R1 release stunned Silicon Valley with frontier-class reasoning at a fraction of the presumed cost — briefly erasing hundreds of billions of dollars in US tech market value. OpenAI soon claimed it had evidence that DeepSeek had used its model outputs in training, in violation of its terms of service. The word "distillation" entered mainstream vocabulary for the first time, carrying an accusatory edge it had never had in the research community.
Act II — Anthropic's "industrial scale" claim (February 2026)
In February 2026, Anthropic escalated the narrative with unusually specific numbers: it said its Claude models had been distilled at industrial scale by DeepSeek, Moonshot, and MiniMax, describing roughly 24,000 fake accounts generating around 16 million exchanges. The claim reframed distillation from an occasional terms-of-service violation into something resembling a coordinated extraction campaign — model outputs harvested like a natural resource.
Both OpenAI and Anthropic explicitly prohibit using their outputs to train competing models. Under their reading, unauthorized distillation is intellectual property theft. Critics counter that these same labs trained on the open internet — including copyrighted material — without asking anyone, which makes the moral high ground feel slippery.
Act III — The K3 accusation (July 2026)
Then came the sharpest escalation yet. On July 16, 2026, Moonshot released Kimi K3 — a 2.8-trillion-parameter open-weight model whose benchmarks reportedly trail only Anthropic's Fable 5 and OpenAI's GPT-5.6. Its mere existence rattled investors and undercut the assumption that export controls had frozen China a generation behind.
Six days later, Kratsios made the US government's first direct, named accusation: that Moonshot had distilled Fable to build K3, using a purpose-built platform designed to rotate access methods and avoid detection. He added a second, arguably more serious allegation — that Moonshot had acquired and accessed Nvidia GB300 Blackwell-generation servers, including via Thailand, in apparent violation of export controls. Treasury Secretary Scott Bessent piled on with the quote of the season: "Open source is not open season on American IP," warning that sanctions and Entity List designations were on the table.
Notably, Kratsios was careful to bless the technique itself, calling legitimate distillation a vital part of the open innovation ecosystem. The objection, officially, is to covert, industrial-scale distillation of proprietary models.
The skeptics push back
Independent researchers immediately questioned the math. Fable 5 had been briefly pulled offline over export-control concerns and only re-released on July 1 — leaving a window of about 15 days before K3's launch. Training a 2.8-trillion-parameter model is not a two-week job, and researchers argued that a fortnight of API access could not plausibly account for K3's capabilities. Distillation may have played some role in some stage of training — it plays a role in nearly everyone's training now — but the claim that K3 is essentially a bootlegged Fable strains credulity for many in the field.
Meanwhile, a coalition of leading tech companies published an open letter urging policymakers not to impose premature restrictions on open-weight models — warning that clumsy rules aimed at Chinese distillation could stifle competition at home and push innovation offshore. As of this writing, no actual enforcement action has been taken: every consequence remains threatened rather than enacted.
The uncomfortable structural truth
Here is the bind that makes this conflict so hard to resolve:
- Detection is genuinely difficult. Black-box distillation looks like heavy API usage. Labs deploy behavioral fingerprints, watermarked outputs, and account-pattern analysis, but a determined actor rotating accounts and access channels (exactly what Moonshot is accused of doing) is hard to prove out.
- Open weights make it worse — or better, depending on your seat. Once a strong model's weights are public, anyone can distill it freely, forever. K3 itself is open-weight; its capabilities are now a global public good, including its potential as a teacher.
- The economics invert the moat. If a $10,000 fine-tune can capture most of the value of a billion-dollar training run, the frontier lab business model — spend enormously, charge for access — faces a fundamental free-rider problem. That, more than any single accusation, is what has Washington and Silicon Valley genuinely worried.
- Compute controls and distillation controls pull in opposite directions. Export controls try to deny China the hardware to train frontier models. Distillation reduces the need for that hardware in the first place. The tighter one screw is turned, the more attractive the other loophole becomes.
What Comes Next
Expect three threads to develop through the rest of 2026:
Technical countermeasures. Frontier labs will invest heavily in distillation detection — output watermarking, canary responses, rate-limiting architectures, and forensic tooling to demonstrate lineage between a suspect model and a teacher. Proving distillation to a legal standard is an open research problem; whoever solves it will shape the policy debate.
Policy formalization. The US is signaling that "distillation attacks" may join chip smuggling as sanctionable conduct. The hard part is drafting rules that punish covert extraction without criminalizing the legitimate distillation that underpins the entire efficient-AI economy — including inside American companies.
The open-weight reckoning. The deepest question is whether frontier capability can be enclosed at all. Every open-weight release resets the global baseline, and every API is a potential teacher. Distillation didn't create that dynamic — it merely made it fast, cheap, and impossible to ignore.
Hinton's 2015 paper described dark knowledge as the hidden information a model carries beyond its explicit answers. Eleven years later, the metaphor has outgrown the mathematics. The dark knowledge in today's frontier models is measured in market capitalizations and national strategies — and everyone, on every side of the Pacific, is trying to distill it.
