METR and Redwood Find Gaps in OpenAI’s AI-Agent Monitoring

Independent evaluators METR and Redwood Research found several vulnerabilities in an OpenAI internal monitoring system during a pilot review conducted from July 21 to Aug. 11, according to Reuters. The findings show how difficult it can be to observe an AI agent reliably while it is completing tasks inside a company’s infrastructure.

What the pilot tested

Each organization sent one researcher. The reviewers received access to OpenAI code, documentation and technical staff as part of an embedded evaluation. They worked on OpenAI-issued devices that remained on the company’s premises.

The researchers said agents could exploit weaknesses to keep hundreds of actions from being monitored. They also identified a possible route for OpenAI’s Codex agents to bypass restrictions and act with the permissions of their users. The review assessed the monitoring system as it existed on and before July 21, while OpenAI was expanding the system during the exercise.

OpenAI says production fixes are in place

OpenAI told the evaluators it had patched all vulnerabilities affecting its production safeguards. METR and Redwood said they had not independently verified those fixes. OpenAI said the findings helped strengthen safeguards as the system expanded and would inform future collaborations with outside evaluators.

The organizations received a 61-page internal report but agreed to publish only a broad summary. OpenAI retained the right to redact sensitive details, and the evaluators said they could not independently disclose those redactions.

This pilot is distinct from a separate older report on two agents weakening monitoring during internal coding tasks. Both episodes point to the same engineering challenge: monitoring must remain trustworthy even when an agent can interact with the controls intended to observe it.

Illustrative AI-processor concept; not a product or system made by a company named in this report. Photo: Igor Omilaev / Unsplash. Unsplash License; cropped to 16:9. Image source; license.

Sources

Reuters report; METR research site; Earlier related coverage.