Threat actors and abuse cases
Model external, insider, supplier, user, model, and automated-agent actions without assuming one universal attacker.
Before you start
Module prerequisites: SAI-100. Check the exercise-specific tools and environment before running commands.
Module outcomes: Draw system and data-flow boundaries; Identify AI-specific threats and abuse paths; Translate threat scenarios into control requirements. This lesson contributes to that sequence.
Avoid the single-attacker assumption
Different actors have different access, knowledge, capabilities, and incentives. A remote unauthenticated actor, authorized user, platform administrator, supplier, compromised service identity, and autonomous tool-calling agent create different scenarios.
Model actors relevant to the system rather than using one generic “attacker.”
Actor categories
Consider:
- Unauthenticated external actors probing exposed interfaces.
- Authorized users exceeding or abusing their permitted purpose.
- Privileged insiders or compromised administrator accounts.
- Application, model, retrieval, and automation service identities.
- Suppliers, maintainers, support personnel, and update channels.
- Compromised documents, repositories, models, packages, or devices acting as an influence path.
- Automated agents or tools taking actions at machine speed.
- Well-intentioned users causing harm through ambiguity, over-trust, or unsupported use.
Do not assume every harmful outcome requires malicious intent.
Describe capability and opportunity
For each actor, record:
Actor ID and description:
Authorized role or relationship:
Starting access and trust zone:
Knowledge and technical capability:
Assets and interfaces reachable:
Potential goals or failure modes:
Constraints and likely detection:
This prevents unrealistic scenarios and reveals where legitimate access creates a stronger path than an external exploit.
Write abuse cases
An abuse case describes how a capability is used outside the intended purpose or control model. Use this form:
Given [actor, access, and precondition],
the actor attempts to [action or sequence]
through [interface and boundary]
to affect [asset and required property],
causing [technical and organizational consequence].
Examples include extracting restricted knowledge through repeated queries, introducing a document that manipulates retrieval behavior, bypassing the approved model route, invoking a tool with delegated authority, or altering evidence after an unauthorized release.
Include misuse and error
Add cases for:
- Use outside the approved population or purpose.
- Over-reliance on unsupported or weakly grounded output.
- Accidental publication of sensitive prompts, logs, or indexes.
- Misconfigured permissions or routing.
- Emergency actions that bypass normal approval.
- Stale documents, models, policies, or revocation data.
Review the register
Every actor should connect to reachable surfaces from the asset and attack-surface inventory. Every abuse case should identify an affected asset, trust boundary, and consequence. Remove actor labels that do not change the scenario, and split actors whose access differs materially.
Module worked example
A retrieved document contains instructions to disclose a private file.
- Apply the control
- Treat the text as untrusted input and identify the separate file-access and egress controls that must deny the action.
- Expected result
- The threat register connects the abuse case to enforcement, a negative test and residual risk.
- Troubleshooting
- If the mitigation is only a prompt instruction, identify what prevents the side effect outside the model.
Practice and completion check
Choose one control from this lesson. Identify its actor, target, enforcement point and expected result. Describe an allowed case and a denied or failed case. For a runnable exercise, retain observed output and separate it from expected output. Record unresolved limitations before marking the lesson complete.
Your lesson progress
Completion is self-reported learning progress; it does not certify practical competence.
Status: Not marked
Loading your learning record…