Introduction to AI threat modelling

Purpose of threat modelling

Threat modelling is a structured way to ask how a defined system could be misused, compromised, manipulated, disrupted, or made to produce unacceptable outcomes. It supports architecture decisions before deployment and remains useful as the system changes.

SAI-110 treats threats as traceable scenarios, not a generic list of AI risks.

Start from SAI-100 artifacts

The minimum inputs are:

• Intended purpose and prohibited uses.
• Actors, owners, and affected parties.
• System context and lifecycle.
• Trust boundaries and numbered flows.
• Sovereignty control requirements and accepted dependencies.

If these are missing, use Define the AI system and Trust boundaries and data flows before continuing.

What makes an AI threat model different

AI systems add probabilistic behavior and new influence paths, but they still depend on familiar software, data, identity, infrastructure, and operational controls. Model both.

Important AI-specific questions include:

• Can inputs or retrieved content manipulate system behavior?
• Can permissions be lost during ingestion, retrieval, memory, or tool use?
• Can model, prompt, adapter, index, or evaluation assets be replaced or poisoned?
• Can output appear authoritative without sufficient evidence?
• Can an agent perform an action outside the user’s or system’s authority?
• Can an attacker exhaust scarce accelerator, context, retrieval, or tool capacity?
• Can evidence be removed or altered to hide behavior?

The SAI-110 method

Use six steps:

1. Define the system, purpose, lifecycle, actors, and boundaries.
2. Inventory assets and attack surfaces.
3. Describe actors, capabilities, goals, and abuse cases.
4. Write end-to-end threat scenarios.
5. Prioritize scenarios and select control objectives.
6. Define verification evidence and record residual risk.

Iterate when a scenario reveals a missing component, flow, asset, or control owner.

Threat-model outputs

The completed model should contain:

• Scope, assumptions, and exclusions.
• Asset and attack-surface inventory.
• Actor and abuse-case register.
• Prioritized threat scenarios with affected boundaries.
• Existing and required controls.
• Positive and negative verification tests.
• Evidence requirements.
• Residual-risk decisions, owners, and review triggers.

The output is not proof that every threat has been found. Its value comes from explicit scope, traceability, review, testing, and controlled change.