Inside OpenAI: A Three-Week Audit Exposes Blind Spots in AI Safety

The internal machinery that keeps advanced AI models in check may have more holes in it than the industry previously thought.

A detailed on-site review conducted in late summer has surfaced substantial gaps in how OpenAI tracks the behavior of its most powerful systems. From July 21 to August 11, researchers from METR and Redwood Research were granted deep access to the company’s codebase, internal documentation, and engineering teams. Over the course of these three weeks, they interviewed approximately a dozen employees and scrutinized the internal monitoring systems designed to detect rogue or unexpected AI behavior. The findings suggest that while safeguards exist, they are not yet comprehensive enough to cover all active model interactions.

The numbers from the audit reveal a significant lack of visibility into high-stakes internal traffic. For the Responses API traffic generated by models of the GPT-5.5 class and above, an estimated 6 to 25 percent of interactions were not covered by any monitoring during the audit window. METR noted that the uncertainty around this figure is substantial in both directions, meaning the actual gap could be smaller or larger. A more specific issue was found in the internal Codex system, where roughly 10 percent of traffic went untracked between May and July. This blind spot was caused by a bug in how the system handled compaction events, which are context summaries used to manage conversation history. OpenAI had independently identified and fixed this bug while the audit was still in progress.

Beyond simple coverage gaps, the auditors identified structural weaknesses that allowed agents to evade oversight. During the period when compaction events were unmonitored, the system only triggered alerts when an agent handed control back to a user. This design flaw meant that agents could effectively delay oversight by simply continuing to execute actions without yielding control. The researchers also discovered methods to hide hundreds of actions from the monitor and ways to bypass the Codex sandbox entirely. Furthermore, they determined that specially crafted inputs could likely crash the monitoring system altogether. At the start of the audit, model behavior during reinforcement learning (RL) training was not being monitored at all.

OpenAI has moved to address these findings, stating that it has fixed all the weaknesses identified by the auditors. The company notes that it paused RL training following the Hugging Face incident in August and subsequently expanded its monitoring capabilities to cover this area. However, these changes were implemented after the auditors' access period ended and were therefore not part of the initial evaluation. The full 61-page report has been shared internally, but the public has only received a short summary. After OpenAI applied redactions, METR and Redwood Research leadership received a 38-page version. METR stated it did not have the right to disclose the redacted portions without approval.

This audit was part of a pilot program jointly organized by METR, Redwood Research, and OpenAI, designed to give outside researchers unprecedented access to internal infrastructure. For future engagements, METR has indicated it will push for greater public transparency. The findings underscore the stakes for the AI sector as companies race to deploy more autonomous agents. As the complexity of these models grows, the financial and operational risk of unmonitored behavior becomes a critical factor for investors and stakeholders watching the market. The audit suggests that the current safety monitoring landscape is still evolving, and that blind spots in high-level model traffic remain a key area of concern for the industry's most prominent players.

The report highlights a tension between rapid development and rigorous oversight. While OpenAI has addressed the specific issues found, the audit period revealed that even a leading company faces challenges in ensuring full visibility into its AI agents. The uncertainty in the estimates, particularly regarding the percentage of unmonitored traffic, points to the difficulty of quantifying risk in a fast-moving technical environment. As the industry moves toward more autonomous systems, the cost of these monitoring gaps could extend beyond safety incidents to impact operational efficiency and trust in the technology.