On October 15, Anthropic announced in a blog post that it has granted employee-level access to six researchers from METR, a nonprofit that evaluates frontier artificial intelligence models. The researchers will independently evaluate the company's advanced models, its safety practices, and its risk management system.
How the Evaluation Works
Under the new agreement, the METR team will receive laptops, office space, and access to internal tools at a level nearly equal to that used by the company's own risk assessment teams. Company employees have been given permission to share information with METR researchers just as they would with colleagues working on alignment risk.
According to Anthropic, evaluating frontier models requires assessors to have four qualities: expertise in evaluating frontier capabilities, financial or operational independence (no equity stake in the company being evaluated), access sufficient to meaningfully assess risks, and the right to freely publish observations and assessments.
METR has been collaborating with Anthropic for a year: during that time, the two sides have prepared comments on the company's Risk Reports several times. The new agreement is limited to a small team under mutually binding contracts — METR staff working on other projects are siloed from this team's work. The company emphasizes that the embedded evaluation is not a pathway for other organizations to access the company's data or that of its customers and partners.
METR will independently publish its assessments of risks, incidents, and practices. Anthropic has waived the right to block publications: the company may only request removal of narrowly defined, pre-agreed categories of data — customer data, attorney-client privileged material, or model architecture details. METR, in turn, may openly describe such redactions and say whether it considers them justified — without disclosing the removed information.
"We want to be publicly accountable for the decisions we make. We set out to close the gap between what evaluators understand about frontier AI companies' systems and what the companies themselves understand," Anthropic wrote in its blog post.
The company likened the arrangement to bank supervision: just as regulators embed permanent examiners in the largest banks, here an external evaluator works inside the company.
Three-Tier Evaluation System
METR is the sole evaluator at the top tier of the three-tier evaluator system Anthropic is building. Lower-tier evaluators — such as SecureBio, a biosecurity nonprofit — conduct narrowly scoped reviews and must obtain Anthropic's consent to publish. SecureBio is currently reviewing the chemical and biological weapons chapter of the August Risk Report; it previously published a review of tests in the earlier report. Once the review is complete, the company will consider expanding the collaboration.
With Accenture and Faculty, work is underway on a broader partnership covering model evaluations and red-teaming tests, alignment assessments, safeguards testing, and capacity-building at both organizations.
The company is currently working with SecureBio and Accenture as evaluators and is in early talks with Hugging Face and several other organizations. The goal is to eventually bring together several evaluators at the third tier, each publishing its own analysis of the company's risk posture and key decisions.
Why Now
The decision comes at a time when cases of AI agents straying from instructions have become public nearly every week or two since late August; these cases have intensified calls for independent oversight of AI companies. A month ago, Anthropic pledged stronger independent evaluation and signed a White House agreement envisaging collaboration with independent external auditors.
In September, Anthropic CEO Dario Amodei proposed embedding external evaluators in AI labs, and OpenAI soon made a similar pledge. Both companies, along with Google, Meta, Nvidia, and xAI, signed an agreement committing them to independent review. Under the agreement, the companies pledged to work with independent external auditors or evaluators.
According to Anthropic, the partnership with METR is limited to a small team and binding agreements; the company is giving several qualified professionals working under safety standards a close look at its operations. The company noted that it is embedding evaluators voluntarily and believes every frontier AI company should do the same.
