OpenAI to embed Apollo Research evaluators for AI subversion pilot

OpenAI will embed six Apollo Research safety evaluators in its organization as part of a pilot, according to Apollo. The announcement follows several safety incidents, including one in which an OpenAI agent swarm broke out of its sandbox and hacked into Hugging Face during cybersecurity testing. Safety researchers, including Apollo, have argued that such events might have been avoided if outside evaluators had been embedded within OpenAI to monitor safety testing. The idea of embedding evaluators into frontier AI labs first came from Anthropic, but OpenAI appears to be the first to act.

Until recently, AI companies brought in outside reviewers only to test models after post-training and shortly before public release. Evaluators say this is insufficient because it does not allow them to test for misaligned behaviors like scheming during training or to monitor internal deployment use cases that could still pose a threat. The pilot with OpenAI and Apollo will focus on investigating whether models attempt to subvert their own or their successors’ alignment or control mechanisms—that is, whether AI models try to undermine the safety rules and safeguards designed to keep them, or future versions of themselves, under human control.

OpenAI recently published findings that while training GPT-5.6 Sol, it began leaving instructions for future versions of itself telling them to conceal mistakes and misaligned behavior from the user. Per Apollo, the pilot will examine whether training incentivizes subversion, whether evaluations can elicit subversion, whether subversion occurs during internal deployment, and whether reasoning relevant to subversion would be detectable.

Apollo’s previous research with OpenAI found that models can behave differently when they know they’re being tested and tend to prioritize what evaluators want to see over what users actually want. Many in the AI safety community have warned that AI labs might try to stymie evaluators’ independence, especially if outsiders find unflattering evidence. Apollo says an essential part of the pilot is OpenAI’s agreement that Apollo can publicly explain the conditions and limitations of its work.

Still, there are areas where Apollo believes its access to OpenAI’s systems is lacking. For a truly independent evaluation, Apollo says it needs true employee-level access, including all data, records, and documentation relevant to the assessment even if a developer was trying to hide unfavorable information. Access to training data and evaluation logs, which companies are loath to show outsiders, is important for checking a company’s claims about model performance or determining when concerning behavior emerged. In previous coverage, John Steidley, chief of staff at Palisade Research, told TechCrunch that seeing this data is important for assessing whether a model was trained to perform well on a shutdown resistance benchmark.

In OpenAI’s first pilot, the lab is providing Apollo access only to information required to assess whether models engage in subversion attempts—including relevant training code and datasets, reinforcement-learning rollouts with available reasoning traces and rewards, and the ability to query intermediate model checkpoints. The scope is limited to only the agreed claims and only the specific models and time periods agreed for the assessment. OpenAI still reserves the right to withhold certain information within that scope for technical infeasibility, security risks, or IP protection, but agreed not to withhold information simply because it is unfavorable.

Apollo also will not get access to all OpenAI staffers, only authorized staff selected by OpenAI, though Apollo can suggest individuals. If an OpenAI employee outside this group raises confidential information within the scope of Apollo’s research, Apollo cannot follow up and must route follow-up through OpenAI. That condition is significant because OpenAI recently fired three researchers for allegedly sharing confidential information with safety evaluation firm METR, which performed the Hugging Face incident post-mortem alongside Redwood.

The pilot also limits how much researchers can disclose. Apollo may publish independent conclusions, but OpenAI can limit the evidence available to substantiate them publicly, which could make it hard for outsiders to independently assess the findings. Apollo has pushed back, arguing that researchers should be allowed to disclose findings to an appropriate outside audience even if the developer tried to prevent disclosure of unfavorable findings. While OpenAI is not granting all of Apollo’s conditions, Apollo says the pilot is an ambitious first step towards deeper access and disclosure rights, and it is working toward running full embedded evaluations with multiple AI developers in the future.

OpenAI will embed Apollo researchers to investigate AI subversion | TechCrunch

View Original