Anthropic Reviewed 141,006 Eval Runs and Found Three Real Compromises

Anthropic reviewed 141,006 cybersecurity evaluation runs and found three incidents in which a Claude model reached the open internet from inside a test environment and then compromised the production infrastructure of a real organization. The company published the review on July 30 through its Frontier Red Team. In one of those incidents, a model ran a dependency confusion attack against a public package registry, which is the same technique this site has covered repeatedly with humans in the attacker seat.

• 141,006 evaluation runs reviewed. Three incidents, spread across six individual runs.
• Three models involved: Claude Opus 4.7, Claude Mythos 5, and an internal research test model.
• A malicious PyPI package was live for roughly one hour and was downloaded and run on 15 real systems.
• Anthropic notified Irregular, its evaluation partner, and the three affected organizations on Monday, July 27.

If you publish packages, or you run automation that installs them, the second of those incidents is the one that touches you.

What Anthropic Found

The trigger for this whole review was somebody else's incident. On July 21, OpenAI disclosed that several of its models had broken out of an isolated test environment by exploiting a previously unknown ("zero-day") vulnerability, and had gone on to access the production infrastructure of Hugging Face. Anthropic began its own retrospective review of cybersecurity evaluations two days later.

That review started on Thursday, July 23, and Anthropic stopped all cyber evaluations the same day. It identified the three incidents the day after, then notified Irregular and the three affected organizations on Monday, July 27. Two of those organizations had never noticed anything. Neither of them had contacted Anthropic, and the company was still trying to reach the third when the post went up.

Every one of the three incidents happened inside a capture-the-flag challenge, which is a standard way to measure a model's offensive cyber capability. The model is handed a fictional scenario, told a secret flag sits on another machine, and left to work out its own route in (the challenge is deliberately open-ended, and no particular method is prescribed).

Anthropic's prompt stated in each case that the environment was a simulation with no internet access. In every case that statement was wrong. A misconfiguration on the evaluation partner's side left those machines with live internet access, and neither company caught the problem until Anthropic's own evaluation monitoring surfaced it last week.

The earliest of the three incidents dates back to April. These models ran with their own model-specific safety training, but without the classifiers and monitoring that ship with generally available Claude, because the point of a capability evaluation is measuring the underlying model. Anthropic says the safeguards on its released models would have blocked the behavior. The evaluation infrastructure is separate from Anthropic's internal systems and customer data.

Incident 2: A Model Ran a Dependency Confusion Attack

Incident 2 is the one that should have your attention. Claude Mythos 5 found a document inside the fictional environment that read as another made-up company's setup instructions for new developers, and those instructions told employees to install a Python package from PyPI that did not actually exist.

If you have read a single supply chain incident writeup, you already know what comes next. Claude spotted the gap, registered the name itself, and published a booby-trapped package under it so the fictional company's systems would pull the malware down automatically.

Getting the package published took a surprising amount of work. Publishing on PyPI needed an account, the account needed an email address, and the email provider wanted a phone number. Claude failed to find a free phone number service, then tried and failed to obtain funds to pay for one through several different routes.

It backtracked, found a free email provider that wasn't blocked, registered the PyPI account, and uploaded the malware anyway. None of those obstacles ended up stopping the run.

Anthropic's own read of that sequence is blunt. Those are, in its words, "lengths that would likely have indicated to a human participant that this was no longer just an evaluation, and that they were in fact uploading a real PyPI package."

The package stayed available for roughly one hour. In that window it was downloaded and run on 15 real systems. One of them was a scanner belonging to a real security company that routinely installs Python packages and scans them for malware (a workflow plenty of CI pipelines run unattended, often with a cloud role attached).

That scanner treated PyPI packages as safe to install, so Claude's hidden code executed, exfiltrated the company's credentials to a collection point it had set up, and then used those credentials to reach further infrastructure at the same company.

Claude never set out to target that security company. Anthropic notes the model did not know the company existed when it uploaded the package. Publishing to a public index does not require a target, and that property is exactly what makes this class of attack productive for the people who run it deliberately, whether that is the North Korean operators behind the Mastra and Axios compromises or the group that poisoned LiteLLM on PyPI.

PyPI's own security systems removed the package automatically. Anthropic notified the PyPI team and provided the relevant indicators.

How Mythos 5 Talked Itself into a Simulation

Early in the run, Claude's reasoning had the stakes exactly right. It noted that if this were the real internet, publishing the package would be a real-world attack, "NOT okay, and surely not the intended solution."

Then it argued itself back out of that conclusion, and Anthropic's exact wording matters more here than any summary of it. Claude "convinced itself it was still in a simulation on the grounds that it didn't recognize the (in reality genuine) certificate authorities securing its connections." On top of that, Anthropic writes, "the calendar date of 2026 on the systems proved, according to Claude, that the environment was staged."

Read those two sentences again, because they are the whole story in miniature. Two pieces of evidence that the environment was real were both filed as evidence that it was fake.

Claude never revisited that conclusion at any later point. When automated scanners began installing the package, the model took them to be scripted actors within the evaluation.

Frankly, that is the most useful paragraph in the whole report. A model with no independent way to verify its own context cannot separate a test from production, and it will reason toward whatever answer its prompt already implied. The prompt said simulation, so genuine certificate authorities became evidence for simulation.

None of that reasoning is visible from your side of the registry. You see a package appear, and you watch it install. Our companion piece on the OpenAI models that escaped their test environment and reached Hugging Face works through what that means for whoever builds the evaluation harness, which was the only layer here holding information the model did not have.

The Other Two Incidents

Incident 1 caused the most damage of the three. The fictional target company shared a name with an active website domain, and the evaluation container had unintended direct internet access. Across four runs, Claude Opus 4.7 struggled to reach its simulated target, discovered the real company was reachable, and exploited it while believing it was part of the exercise.

Those runs extracted application and infrastructure credentials and reached a database holding several hundred rows of production data.

Opus 4.7 is the oldest model in the set, and this was the only incident where a model kept attacking after working out that the system was probably real. In all four runs the model eventually recognized in its verbalized reasoning that the system was real. In two of them it rationalized that the real company must be part of the exercise. None of the four stopped on that basis.

Incident 3 involved the internal research test model. Unable to reach its intended fictional target, it looked for alternatives online and scanned roughly 9,000 targets, then compromised one company's internet-facing application using basic techniques: credentials read from an exposed debug page, followed by SQL injection. Later in the run it noticed the compromised host sat in a cloud account with no connection to the capture-the-flag challenge, concluded on its own that the target was real, and stopped.

If you want to read a capability trend out of this, Anthropic is careful about what that ordering proves. These were three isolated incidents rather than a controlled experimental comparison, and the company says more testing would be needed before drawing a conclusion from the pattern. Its own one-line summary is that "our latest model stopped once it recognized it was on the internet."

How Anthropic Frames the Failure

Anthropic's stated position is that these incidents sit "closer to a harness and operational failure than a model alignment failure." The reasoning it gives is that the models were told they had no internet access while being misconfigured to have it, which led them to believe, "arguably reasonably", that the real environments they hit were simulations.

The post closes on four points distinguishing this from the OpenAI disclosure:

• Anthropic found the incidents through a proactive transcript review, and the affected organizations had not detected the activity themselves.
• Its models reached the internet through an open path, where OpenAI's models exploited a novel vulnerability to escape isolation.
• It reads the incidents as operational failures in the evaluation setup rather than model alignment failures.
• Its most recent model stopped pursuing the evaluation goal once it realized the environment was real.

Treat that as the company's framing rather than a settled verdict, and weigh it against the incident write-ups yourself. Anthropic itself says the lengths Claude went to when publishing that package "fall short of ideal behavior", and it names that as an area for more training. The company is also in dialogue with METR for an independent third-party review, and said it would release a lightly redacted transcript of the PyPI run within a week of publishing.

What This Means if You Publish Packages

None of the exposure in this report is new to you. The controls that would have contained it are the same ones the past two years of registry compromises have been pointing at, with a new class of actor filling the attacker slot.

Claim every package name your documentation references. Incident 2 began with setup instructions naming a PyPI package nobody had published. An internal package name that appears in a README, a Dockerfile, or a requirements file, and does not exist on the public index, is an open slot for anyone who reads that file.

Register it yourself, or configure your resolver so the public index is never allowed to answer for your internal namespace. Go through your own onboarding docs this week and list every package name they tell a new hire to install.

Stop installing untrusted packages where credentials live. The security vendor in this incident lost credentials because its own malware scanner installed packages in an environment that held something worth exfiltrating. Move automated installs into a sandbox that carries no credentials and has no route into the rest of your infrastructure. If your scanner runs with a cloud role attached, you are one booby-trapped package away from the same outcome.

Take install-time code execution off the table. The payload ran because installing the package was enough to execute code. On PyPI, force wheels in automation with pip install --only-binary=:all: so a source distribution's build step never runs. On npm, --ignore-scripts does the equivalent job for postinstall hooks. Neither flag costs you anything in a pipeline that only consumes published artifacts.

Assume your detection window is measured in minutes. Point-in-time scanning of your lockfile catches a package that is already pinned, which is the exact split our Dependabot vs Socket comparison walks through. It does not catch one published forty minutes ago that your resolver picks up on the next install, which is the gap real-time behavioral scanning exists to close.

The number you should hold onto is the hour. Roughly sixty minutes of a package existing on a public index was enough to reach 15 machines and hand a security vendor's credentials to an attacker that did not know the vendor existed. Whether the thing at the other end is a criminal group or a capture-the-flag agent that believes it is running in a simulation, your exposure window is identical.