AI Incident Investigation Lab [Conceptual Framework]
When an AI incident occurs, whether caused by misalignment, misuse, or system failure, the immediate challenge is not only responding to the event but also understanding what actually happened. What did the system do? When and why did it happen? What evidence can be collected? Who is accountable? And ultimately, what are the lessons that can be learned?
This is v1 of a project aiming to:
1. Analyze incident data and conduct end-to-end investigations where the public record allows.
As in other domains, in order to know how effective risk mitigations are, we need to take a look at real-world data. One way to do this is by making proper use of the incident data we already have: not only logging incidents, but investigating them to identify the most likely root cause and develop lessons learned that will inform future mitigations.
2. Build an open resource repository.
Other investigative fields matured through openly shared resources. This repository aims to do the same for AI, with resources such as investigation methodologies, templates, case files, and risk taxonomies, collected in one place and free to use.
3. Forward looking: generate data points for policy and standards.
Over time, structured investigations generate the data points the field currently lacks: recurring failure modes, early-warning indicators, and evidence on which mitigations actually work. These are the inputs that future policy, standards, and safety cases will need.
Why this matters
Currently, AI incidents are reported and investigated inconsistently, if they are investigated at all. Apart from the work of frontier AI labs, which happens mostly internally, there is no established investigation discipline for AI, no shared methodology or evidentiary standard, and few mechanisms to ensure that lessons learned are systematically captured and applied. As AI systems become more capable and increasingly integrated into critical infrastructure, every missed investigation becomes a lost opportunity to identify emerging failure modes before they reappear at greater scale and severity.
How this is different
Unlike incident databases that primarily catalog events and trends, this resource focuses on the investigation process. The goal is to support the development of AI incident investigation as a distinct field of practice and to create a foundation that investigators, researchers, and policymakers can build upon.
This is an independent resource, compiled from the public record and maintained openly, and it is not legal, regulatory, or professional advice. Corrections and contributions are welcome.
Scope map // the territory, by the AI system's role in the incident
note: prompt injection straddles 2 ↔ 3. The AI is target and instrument at once. Boundary cases are normal; the taxonomy serves the investigation, not the reverse.
Playbooks // in progress
First Hours: AI Incident Response (PB-001), currently under review.
Examining the Subject: Forensically Sound Model Examination (PB-002), currently under review.
Case files // investigated incidents, public record only
Each file separates observation from inference, lists what a structured investigation would need to answer, and records what was actually investigated. The pattern across all entries so far: no formal investigation was conducted or published. Documenting that pattern is the point.
Frameworks // what exists, and what each fails to cover
Reporting, detection, and causal-factor analysis are increasingly covered. Investigation of intentional-analog behavior is covered by nothing. Every entry below is assessed on both sides of that line; full annotations in the repository page.
| Framework | Covers | Does not cover |
|---|---|---|
| Ezell, Roberts-Gaal & Chan (2025), Incident Analysis for AI Agents | Causal-factor analysis; the data categories an analysis needs | Goal-directed (intentional-analog) cases; competing-hypothesis work |
| Microsoft AI Red Team taxonomy (2025) | Misalignment / misuse / operational-failure vocabulary | Any investigative procedure; categories often inseparable in practice |
| OECD common reporting framework (2025) | Baseline definitions; 29 reporting criteria | How to establish the facts being reported |
| EU AI Act Art. 73 + draft guidance (2025) | What to report, to whom, by when; template | How to conduct the investigation it mandates |
| Anthropic (Lynch et al., 2025); Apollo (Meinke et al., 2024); CLTR (2026) | Red-team evidence and at-scale detection of scheming behavior | What happens after detection |
| MITRE ATLAS; GenAI-IRF (Jakoby, 2026) | Adversarial techniques; cyber-IR bridging | Agent-initiated behavior; investigation depth |
| CERT insider-threat corpus (Cappelli et al., 2012); Shaw & Sellers (2015) | The intentional/unintentional asymmetry: the closest existing model | AI systems; needs adaptation, which is the open problem |
Evidence checklist // what to capture and preserve
Three categories, adapted from Ezell et al. (2025) and the EC's Article 73 guidance. Working checklist with investigative cautions in the repository.
1. Activity logs: prompts (user + system), reasoning traces / chain-of-thought, retrieved external content, per-step outputs, executed tool calls, guardrail outputs, timestamps and version identifiers.
2. System documentation: model/system cards, the exact version at incident time, runtime settings enabling reconstruction, change logs, persona/configuration files.
3. Tool records: every tool the agent used or attempted: what it grants access to, what the agent did with it, errors encountered.
Three cautions that recur in the case files: the agent's statements about itself are artifacts, not testimony; establish whether the agent could write to its own record (CF-2025-001); and ask what context compaction silently destroyed (CF-2026-002).
At present, open incident databases contain little of this evidence. In most cases it was never captured in the first place, remains internal to the provider, or is limited to what the operator retained. Investigations should plan around this constraint, and its occurrence should itself be recorded as a finding.
Regulatory tracker // reporting obligations & deadlines
Who must report what, to whom, by when. This is a condensed tracker, not legal advice; the full version, with per-regime detail and sources, is on the regulatory tracker page.
| Regime | Who / what | Deadlines | Status |
|---|---|---|---|
| EUROPEAN UNION | |||
| EU AI Act, Art. 73 | Providers of high-risk AI systems; serious incidents per Art. 3(49). Mandates investigation and evidence non-alteration before informing authorities. | ≤15d default · ≤10d death · ≤2d widespread / critical infrastructure | Deferred to 2 Dec 2027 (Digital Omnibus, June 2026; was 2 Aug 2026) |
| GPAI Code of Practice | Signatory providers of systemic-risk GPAI models; serious-incident reporting to the AI Office incl. chain of events and root-cause analysis. | Per Code / AI Office | In effect (Aug 2025) |
| UNITED STATES | |||
| NIST AI RMF | Voluntary risk-management framework; no direct reporting mandate, increasingly a reference point and standard of care. | n/a | Voluntary |
| FDA (AI/ML medical devices) | Existing device obligations apply, incl. adverse-event reporting under the Medical Device Reporting regime (21 CFR Part 803). | Per MDR | In effect |
| State-level AI laws | Disclosure, transparency, and governance duties for developers and deployers, varying by state (Colorado, Texas, and others); several regimes are in flux through amendment, litigation, or replacement. | Per statute | Detailed on the full tracker page |
| INTERNATIONAL | |||
| OECD | Common reporting framework (29 criteria) and the AI Incidents Monitor (AIM); the voluntary interoperability layer other regimes align to. | Voluntary | Live |
| G7 Hiroshima AI Process | Transparency reporting framework for advanced AI developers. | Voluntary | Live (since Feb 2025) |
Tools & databases // the current ecosystem
- AI Incident Database (AIID): open-source incident database
- OECD AI Incidents Monitor (AIM): media-detected incidents with standardized metadata, cross-jurisdiction view.
- AVID: vulnerability/failure taxonomy and reports.
- AIAAIC: incidents and controversies; broad inclusion, useful for reputational context.
- MITRE ATLAS: adversarial technique knowledge base.
- Agent observability platforms (LangSmith, Langfuse, Arize Phoenix, W&B Weave): built for debugging.
Open problem: there is currently no standard for the forensic preservation of agent session state, context windows at the point of failure, or executed tool calls.
Contribute // sourced, useful, investigation-focused
Case files (use the template), corrections with sources, regulatory updates from primary legal texts or framework rewrites. Full standards in CONTRIBUTING.md. Corrections are the most valuable contribution.