Anthropic announced that it has begun giving employee-like access to a team from METR, a nonprofit that evaluates frontier AI models, and laid out a three-tier framework for working with independent evaluators. The move follows Anthropic‘s pledge, as part of the White House Accord on Super Intelligence, to partner with independent external auditors or evaluators. METR will publish the results of its risk assessments; over the past year it has already worked with Anthropic on several reviews of its Risk Reports. Anthropic is also working with SecureBio and Accenture as evaluators and is in early discussions with other organizations, including Hugging Face.

The rationale for the program is that more powerful models require stronger safeguards and more challenging risk assessments, and that public accountability conflicts with the need to keep some safety-relevant information private, either because it could help malicious actors or because it is commercially sensitive, such as compute or training details. Anthropic‘s approach is to give third-party evaluators deep access and have them publish evidence and reasoned conclusions, aiming to close the gap between what evaluators understand about a frontier AI company’s systems and what the company itself understands. The post draws an analogy to bank supervision, where regulators place resident examiners inside the largest banks because outsiders otherwise know less about risk positions than the banks do.

Anthropic lists four requirements for evaluators. They need expertise in assessing frontier model capabilities, including an understanding of how evaluations can go wrong, such as alignment faking. They need independence: no equity in the company being evaluated, and no financial or operational dependence on it. They need enough access to meaningfully evaluate risks, including visibility into how models were trained and how safeguards function, beyond access to the latest models. And they need freedom to publish observations and assessments of a model or process.

The METR engagement is the deepest tier described. Six METR employees have been onboarded with company hardware, office space, and access to internal tools and systems roughly on par with Anthropic‘s own risk assessment teams. Outside a small number of clearly defined categories, such as data about customers, attorney-client-privileged information, and information about model architectures, Anthropic employees are encouraged to share information with them as they would with a colleague working on alignment risk. METR can publish its analysis of risk levels, incidents, or practices; Anthropic cannot veto publication, but can require redaction of information falling into the narrow pre-defined categories, and METR may publicly characterize whether it believes the redactions were justified. The engagement is limited to a small team under enforceable agreements, and other METR employees are siloed from that team’s work.

Anthropic is developing three tiers of access for evaluators. At the first tier, evaluators conduct narrowly scoped risk investigations, receive access determined by Anthropic, and can publish findings only with approval. At the second tier, evaluators have more information and can publish without approval, although Anthropic can still require removal of specific categories of information and evaluators can state their views on whether redactions were justified. The third tier is the METR arrangement: employee-like access with publication rights similar to the second tier. SecureBio is currently in the first tier, reviewing the chemical and biological weapons chapter of Anthropic‘s August Risk Report after completing and publishing a pilot review of the previous report; Anthropic says it will consider moving SecureBio to projects with increased access afterward. With Accenture and Faculty, Anthropic is working toward a partnership that includes evaluating and red-teaming models, conducting alignment assessments, testing model safeguards, and building capabilities across both organizations. Other partnerships remain early-stage, with more limited information and publication rights, and Anthropic says it hopes to eventually have several evaluators at the third tier, each publishing their own analysis of its risk posture and key decisions.

Anthropic emphasizes that it is embedding evaluators voluntarily, and states its belief that every frontier AI company should do the same.