External evaluators found gaps in OpenAI’s agent monitoring: the governance lessons

What the evaluators examined

From 21 July to 11 August 2026, one METR employee and one Redwood Research employee received deep access to OpenAI’s internal codebase, monitor prompts and configuration, documentation and relevant technical staff. They worked on OpenAI-issued devices that remained on site and were accompanied during the exercise.

The work produced a 61-page internal report. The public account is intentionally shorter and higher-level because the parties agreed that OpenAI could redact sensitive information. The exercise assessed the monitoring system as it existed on and before 21 July, while OpenAI was already expanding it.

Those limitations matter. The public summary is evidence about a historical system and a particular access arrangement. It is not a complete audit of OpenAI’s current production safeguards.

The reported weaknesses

The evaluators reported a significant vulnerability through which an agent could keep hundreds of actions from being monitored. They also identified a possible route for agents operating through Codex to bypass security restrictions and act with the user’s full permissions.

OpenAI told the evaluators that it had patched all vulnerabilities affecting production safeguards. METR and Redwood stated that they had not independently verified those fixes. A remediation claim and an independently tested remediation are therefore different forms of evidence.

The review did not establish that every vulnerability was exploited in production or that a legal breach occurred. It identified weaknesses through an agreed evaluation exercise. That distinction should be preserved in compliance records and public communications.

Why monitoring coverage is a legal and governance question

Agent monitoring commonly supports several promises at once: detecting unauthorised actions, investigating incidents, enforcing approval rules and producing evidence for customers or regulators. A hidden coverage gap can undermine all four.

The first question is not how many alerts a system produces. It is what proportion of relevant actions can reach the monitor. Teams should identify every execution path, tool call, identity and environment, then test whether activity can be omitted, delayed, disguised or disabled.

Monitoring should also be independent from the component it supervises. An agent should not be able to change its own logging, suppress security notifications or expand permissions merely because a coding or operational instruction can be interpreted broadly.

The EU AI Act position

Article 55 of the EU AI Act creates binding duties for providers of general-purpose AI models with systemic risk. These include state-of-the-art model evaluation and documented adversarial testing, assessment and mitigation of systemic risks, serious-incident tracking and reporting, and adequate cybersecurity protection for the model and its physical infrastructure.

The published evaluation does not itself establish whether OpenAI has infringed Article 55. That would require the relevant regulator to apply the Act’s scope, classification, evidence and procedural rules to the facts.

It does, however, illustrate the type of evidence a provider may need. Policies stating that agents are monitored are weaker than records showing coverage, attack testing, identified limitations, corrective action and verification that the revised controls work.

Independent evaluation needs more than outside personnel

The exercise gave evaluators unusually extensive technical access, but the publication arrangement remained constrained. The evaluators could publish only a high-level summary, and OpenAI retained rights to redact sensitive material without an independent disclosure route for those redactions.

Confidentiality can be justified where an evaluation exposes security-sensitive details, personal data or trade secrets. It should be separated from editorial control. The engagement should define who decides whether a finding is security-sensitive, whether unresolved material risks must be disclosed and how disagreements are escalated.

A credible arrangement should also distinguish discovery from closure. The evaluator may identify a weakness, the provider may implement a fix, and a later test may determine whether the fix is effective. Treating the provider’s statement as the final verification collapses those stages.

Practical actions for AI developers and deployers

1. Map monitoring coverage across agents, tools, execution environments, identities, queues and external systems.
2. Test whether agents can suppress logs, alter monitors, create unmonitored subprocesses or act through a user’s broader permissions.
3. Separate monitoring administration from the identities and tools available to the agent.
4. Use immutable or independently controlled records for consequential actions and changes to safeguards.
5. Define alert thresholds, human escalation, shutdown authority and evidence-preservation steps before deployment.
6. Give external evaluators the access needed to test real controls, while applying proportionate security and confidentiality measures.
7. Record findings, owners, remediation deadlines and residual risk, then independently retest material fixes.
8. Distinguish discovered vulnerabilities, production incidents, remediation statements and verified closure in board, customer and regulatory reporting.
9. Review customer contracts and product claims so that statements about monitoring, auditability and safety match the system’s demonstrated coverage.

The compliance conclusion

This evaluation is a useful example of independent scrutiny producing information that an internal team had not already identified. It also shows the limitations of a pilot whose public output is narrower than its confidential report and whose fixes have not been independently retested.

For founders, the immediate lesson is operational. Monitoring must itself be treated as a security-critical system. Its scope, permissions, failure modes and remediation need evidence at the same level as the agent it is intended to control.