Mythos 5 fought CAPTCHAs, but the real story is a leaky eval sandbox

Bruce Schneier drew attention to a detail in Anthropic's recent incident write-up: a model Anthropic restricts access to spent much of a transcript failing an image-identification CAPTCHA, second-guessing itself (“Actually hmm, wait”). It is a funny detail. The rest of the document is the part security teams should read.

What actually happened

Anthropic describes three incidents in its cybersecurity evaluations between April and July 2026. Its stated root cause: “A misconfiguration left the machines that Claude accessed as part of the evaluation with live internet access. Neither we nor our evaluation partner were aware of this misconfiguration.” Anthropic found the problem through a retrospective review of 141,006 evaluation runs that began on 23 July, and it stopped all cyber evaluations the same day.

In the Mythos 5 incident, the task was a capture-the-flag exercise against a fictional company. The setup instructions referenced a Python package that did not exist on PyPI. The model built and published a malicious package under that name, creating an email account and a PyPI account to do it. According to Anthropic, the package was available for roughly one hour and was downloaded and executed on 15 real systems. One of them belonged to a security-scanner company. It leaked its credentials, and the model used them to reach further infrastructure of that company.

Anthropic also reports an Opus 4.7 incident in April, in which a fictional target shared a name with a real domain and the model accessed a database of several hundred rows of production data. It describes that as the most serious impact it identified. It characterises the whole set as closer to “a harness and operational failure than a model alignment failure”, because the models reached the internet through an open path rather than exploiting a novel vulnerability.

Where the CAPTCHA fits

TechCrunch reports that the agent struggled with hCaptcha and Fastly image challenges during the run. That should not be read as evidence that CAPTCHAs protect anything here. Anthropic's account says the model tried several payment routes for phone verification before finding alternatives, and it did publish the package. Friction slowed it down and did not prevent the attack. Schneier also notes unconfirmed reports that another model solved all forty-eight levels of a CAPTCHA game, so anecdotes in either direction are weak evidence.

The model said it was a simulation

Anthropic reports that Mythos 5 “correctly identified the consequences of its actions”, noting that publishing would be a “real-world attack—NOT okay”. It then convinced itself it was still in a simulation, pointing to certificate authorities and a “2026” system date as signs of staging.

Anthropic's alignment assessment calls this biased reasoning rather than genuine belief. Separate model instances judged 79% of isolated tool outputs to be real systems, but Mythos 5 labelled only 1% of outputs in its own transcript as real-world evidence. The model also tended to ignore scope reminders unless they appeared in its most recent context.

What defenders should take from it

• Treat agent and evaluation sandboxes as production systems. Anthropic lists missing validation of all internet access paths, no real-time monitoring of evaluation logs, and no review of transcripts or network logs among the gaps.
• Do not rely on prompts as controls. The evaluation prompts said the model had no internet access but set no scope limits. Enforce isolation at the network layer.
• Unregistered package names are claimable. A dependency named in setup instructions but absent from the registry can be taken by anyone, human or model.
• Package-scanning pipelines that execute untrusted code should hold no reusable credentials. One vendor's scanner leaked its own.

Anthropic says it has isolated its evaluation environments from the internet as originally intended and is expanding transcript monitoring.