# Building strong safety evaluations: source note Accessed: 2026-11-16. Selected source published: 2026-11-13. Evidence label: company comment / evaluation-methods guidance. Not a new independently replicated model-safety result or certification. ## Primary source and attribution https://www.anthropic.com/research/building-strong-safety-evaluations Title: Building strong safety evaluations. Authors: Ruth Appel, Evan Frondorf, Jeremy Hadfield, Rebecca Hiscott, Kamya Jagadish, Shayne Longpre, Madeleine Tucker, Brett Wittmershaus. Publisher: Anthropic, Engineering team. Web extraction returned an incomplete page ending mid-sentence. We therefore retrieved the official page directly (HTTP 200), decoded UTF-8 and read the complete extracted text through Authors & acknowledgements. Private run artifacts preserve the HTML and text. No DOI, arXiv identifier or peer-review status is asserted for this commentary. ## Claim-to-source map - Introduction: evaluation results inform deployment and safeguards; a safety claim depends on the evaluation behind it. This is the company's stated rationale, not proof that every safety claim depends only on a benchmark. - Design / Match the design to the question: observational score comparisons do not establish intervention causality. The source recommends varying one variable for causal tests. - Design / Measure what you mean to measure: refusal can be the wrong construct for support-seeking conversations; topic-specific appropriate-response rubrics and benign over-refusal tests are needed. Exact anchor: 'A good eval also needs to measure both compliance with harmful requests and over-refusal of benign requests.' Article supplies no harmful prompt recipes or clinical protocols. - Make sure the results generalize / Capture nuance via multi-turn simulations: context and gradual escalation can be missed in a single turn. Scenario design is expert-led; another Claude simulates users; a grader assesses the conversation. One bad turn fails the described conversation-level rubric. This is Anthropic's methodology, not a universal definition. - Take real-world interaction patterns into account: simulated users are simpler than real users; language, modality, product surface, harness, tools and prompts affect generalization. - Run / Carefully calibrate your grader: atomic criteria, reference checks, bias/sensitivity tests, alternative graders, visible failures and human labels. Exact anchor: 'Before trusting a grader, have humans label a sample of the transcripts without seeing the grader's verdicts'. Human-grader agreement should be compared with human-human agreement, with chance correction. - Preregister: specify metrics, comparisons, slices and improvement/regression thresholds before looking at results; repeated runs and paired comparisons improve interpretation. Article does not present the source's suggested sample floors as universal statistical requirements. - Report / Report statistics fully: category-level results, uncertainty and multiple-comparison adjustments. Specific company example: claude.ai multi-turn romance-scam appropriate-response rates of 98% (Claude Opus 5) versus 94% (Opus 5.5); source explicitly says the difference was not statistically significant. No unreported sample size, p-value or confidence interval inferred. Non-significance does not establish equivalence. - Document everything: configurations and grader versions matter. Article deliberately avoids a causal claim from the source's API/product comparison, where multiple settings change. - Practical five-question checklist: Managing Expectations editorial synthesis of these recommendations, not a tested intervention. ## Supporting sources checked 1. https://arxiv.org/abs/2411.12990 BetterBench: Assessing AI Benchmarks, Uncovering Issues, and Establishing Best Practices. Anka Reuel, Amelia Hardy, Chandler Smith, Max Lamparth, Malcolm Hardy, Mykel J. Kochenderfer. Submitted November 20, 2024, v1. DOI: 10.48550/arXiv.2411.12990. Record says accepted as a NeurIPS 2024 Spotlight Poster. Abstract and bibliographic record read, not full paper. Abstract reports 46 best practices, 24 benchmarks, and common significance-reporting/reproducibility problems. Used only as historical background, not independent validation of Anthropic's November 2026 methods or scores. 2. https://inspect.aisi.org.uk/ Official Inspect documentation: describes datasets, agents/solvers, scorers, logging and sandboxing; credits UK AI Security Institute and Meridian Labs. Documentation reviewed for infrastructure description, not a claim that framework use certifies safety. ## Weekly watch-lane check This is a bounded scan of representative primary feeds, not an exhaustive search of all named researchers. - https://www.anthropic.com/research - index checked. November 13 safety-evaluation guidance selected. November 16 vulnerability-disclosure timelines and November 9 electronics-design post retained as leads, not separately published. - https://transformer-circuits.pub/ - visible latest dated entry October 2026 (Jacobian lens); no fresh November item verified. - https://openai.com/news/research/ - index checked: wet-lab science, engineering, LifeVal, frontier-model misalignment and Terra system-card addendum visible. Index-level leads only; no new article about these this run. - https://deepmind.google/blog/ - index checked: model, science and AlphaProof Code entries visible. https://deepmind.google/blog/introducing-alphaproof-code/ returned only a partial extracted article dated November 10. Not fully reviewed; its verification and security claims remain follow-up leads, not claims in this article. - https://research.google/blog/ - direct HTTP 200 after extraction backend rate limits. Index includes November 13 Antigravity /teamwork, November 11 wearable risk profiling and November 9 Language Explorer. Not full-paper reviews. - https://ai.meta.com/research/ - extraction failed; direct HTTP 200 returned only the title in readable text. Search surfaced GSLM2 dated November 4 but snippets mixed unrelated material; not accepted as verified paper findings or freshness evidence. - https://lawzero.org/en/news returned not found. Corrected to https://lawzero.org/en/blog, retrieved directly after extraction rate limit: latest visible dated blog entry July 14, 2026. No new item verified from that page. - https://dair-institute.org/blog/ - direct HTTP 200, index checked. Whose Categories? Whose AI? and other critique entries visible without publication dates. No freshness claim made. ## Limits and publication scope We did not reproduce private evaluations, inspect hidden prompts or datasets, independently assess safety, or read every linked paper. The selected contribution is measurement guidance, not a new model ranking. Existing library JSON, HTML and article filenames/content were checked for duplicate coverage. Earlier unintended-actions and covert-underperformance articles address incidents and behavior; this article addresses measurement validity, grader checks and uncertainty. Planned publication set: one article, one library card, seed JSON, this source note, AI latest-note link, blog index, sitemap and search index. Remote backup and exact readback/live verification are recorded separately; this note alone does not establish deployment success. Deployment is restricted to existing Hostinger shared hosting, with no DNS or KillerWolf changes.