Meta researchers have developed a method for detecting distilled AI models that identified 79% of tested copies at a false-positive rate of 0.1%, without inspecting the suspect models’ internals. The technique, called WM-Basin , targets models trained on outputs from watermarked “teacher” models and appears in a paper published on October 27, 2026.
Key takeaways
WM-Basin examines API-generated text rather than model weights or logits.
Meta evaluated the technique across 1,200 student models.
Detection survived several model modifications but weakened with clean-data fine-tuning and paraphrasing.
How WM-Basin detects distilled AI models
WM-Basin uses a separate, clean probe model to analyze text generated by a suspected distilled model. Researchers from Meta FAIR and Meta Superintelligence Labs describe the approach in “Watermark Basins: Black-Box Detection of Distilled LLMs,” published under arXiv:2610.37266.
The paper examines a specific copying scenario: someone gathers responses from a watermarked teacher through its API, then uses those responses to train a student model. WM-Basin looks for evidence of that training relationship using only the student’s generated text.
The detector passes that text into a fixed probe model and converts it into gradient representations —a way to examine how training on the text would move the probe’s parameters. It then distinguishes ordinary fine-tuning directions from alignment with a key-specific “watermark basin” in parameter space.
This makes the technique a form of black-box analysis: access to the suspect model’s weights or logits is unnecessary . Text responses obtained through an API are enough for the detection process.
A 79% detection rate across 1,200 student models
According to the researchers, WM-Basin reached a 79% true-positive rate while keeping the false-positive rate at 0.1% when n-gram=1 was used, whereas the strongest prior black-box token-level detector managed only a 29% true-positive rate under the same conditions.
The authors further note that as detection thresholds become stricter and n-gram windows grow larger, WM-Basin’s lead over competitors widens.
The evaluation included 1,200 student models , ranging in size from 135 million to 8.2 billion parameters. The researchers trained these models on mathematical reasoning traces drawn from OpenR1-Math, spanning six families of student architectures, multiple teacher models, and various watermarking approaches.
These findings underpin the paper’s claimed gains in detecting distilled AI models within its mathematical-reasoning assessment.
Which modifications preserve or weaken detection?
The researchers found that WM-Basin’s detection signal survived quantization, pruning and reinforcement learning after distillation. Quantization compresses a model to reduce running costs, while pruning removes parameters.
The method also remained more reliable when token-level signals weakened, including during low-temperature sampling.
Other changes reduced its effectiveness. Extensive fine-tuning on clean, unwatermarked data degraded detection, while paraphrasing the teacher’s reasoning traces provided another potential route for evasion. The authors specifically identify continued supervised fine-tuning on unwatermarked material among these approaches.
Watermark evidence for AI developers
The combination of higher detection and a low false-alarm rate potentially strengthens watermark-based evidence in legal or contractual claims about model copying. For that use, the false-positive rate is especially relevant because it measures erroneous flags rather than successful identifications.
WM-Basin’s reported comparison pairs the same 0.1% false-positive rate with a 79% true-positive rate, versus 29% for the earlier token-level method. The underlying tests used student models trained on mathematical reasoning traces from OpenR1-Math .
Article produced with the assistance of artificial intelligence and reviewed by the editorial team.
