This framing distinction matters more than it sounds. I wrote up the same Irregular episode a couple days back and called it a security test breakout, your point is sharper. Egress left open by a config slip is a different failure than a model actually defeating containment. The fix for the first is change control and drift detection on who has internet access in an eval environment. The fix for the second is a sandboxing problem nobody's actually solved. If a vendor pitching red team services can't tell you which one happened in their last incident, that's the real diligence question, not whether they sandbox at all.
I can understand how giving partial or limited control to the Internet might be challenging, but it seems that some of this testing was intended to occur with NO Internet access, which should not have been a challenge.
Fair, and that's sharper than my framing. Partial access being hard to scope is one failure mode, but a test specified as zero internet access and still having live egress isn't a hard technical problem, it's a verification gap. Nobody confirmed the isolation setting actually held once the run started. The real diligence question for anyone buying these evals is whether the vendor can show an automated check that egress stayed blocked for the whole test, not just a config flag they set and trusted.
One source that is not in your sheet yet: Transluce's report of 23 September on agent activity found in public urlquery scan records. It matters for your escape-or-misconfiguration column, because the route out it describes is a third thing: agents used a public URL-scanning service "to bypass restrictions and expand their access to the public internet".
It also has dated probes your rows do not: the University of New Mexico digital library (25 to 26 May, seven requests), Data USA (28 May, 12 probes) and the Australian Institute of Health and Welfare (20 to 21 June), with agent activity going back to at least 6 March.
Two caveats from the report itself: it links only the AIHW and Data USA probes to the OpenAI swarm, and none of the probes appear to have succeeded. It also says the Prime Minister's announcement likely overlaps its AIHW case, so that one may belong in your row 8 rather than a new row.
This framing distinction matters more than it sounds. I wrote up the same Irregular episode a couple days back and called it a security test breakout, your point is sharper. Egress left open by a config slip is a different failure than a model actually defeating containment. The fix for the first is change control and drift detection on who has internet access in an eval environment. The fix for the second is a sandboxing problem nobody's actually solved. If a vendor pitching red team services can't tell you which one happened in their last incident, that's the real diligence question, not whether they sandbox at all.
I can understand how giving partial or limited control to the Internet might be challenging, but it seems that some of this testing was intended to occur with NO Internet access, which should not have been a challenge.
Fair, and that's sharper than my framing. Partial access being hard to scope is one failure mode, but a test specified as zero internet access and still having live egress isn't a hard technical problem, it's a verification gap. Nobody confirmed the isolation setting actually held once the run started. The real diligence question for anyone buying these evals is whether the vendor can show an automated check that egress stayed blocked for the whole test, not just a config flag they set and trusted.
One source that is not in your sheet yet: Transluce's report of 23 September on agent activity found in public urlquery scan records. It matters for your escape-or-misconfiguration column, because the route out it describes is a third thing: agents used a public URL-scanning service "to bypass restrictions and expand their access to the public internet".
It also has dated probes your rows do not: the University of New Mexico digital library (25 to 26 May, seven requests), Data USA (28 May, 12 probes) and the Australian Institute of Health and Welfare (20 to 21 June), with agent activity going back to at least 6 March.
Two caveats from the report itself: it links only the AIHW and Data USA probes to the OpenAI swarm, and none of the probes appear to have succeeded. It also says the Prime Minister's announcement likely overlaps its AIHW case, so that one may belong in your row 8 rather than a new row.