Observability and operational evidence
Design signals that explain service health, AI behavior, policy outcomes, change, and incidents without leaking sensitive content.
Before you start
Module prerequisites: SAI-100, SAI-120. Check the exercise-specific tools and environment before running commands.
Module outcomes: Map controls to technical evidence; Define audit events and evidence retention; Verify model, container, configuration, and decision provenance. This lesson contributes to that sequence.
Observe the complete AI service
Infrastructure health alone cannot explain an AI system. Operators need correlated signals across request handling, identity and policy, retrieval, model inference, tool execution, user outcomes, change, and evidence production.
Design telemetry around questions the team must answer:
- Is the service available within its approved capacity and latency envelope?
- Are users receiving grounded, permitted, and useful outcomes?
- Which model, prompt, corpus, policy, tool, runtime, and configuration produced a result?
- Are denies, redactions, escalations, no-answer outcomes, or unsafe attempts changing?
- What changed before an incident or quality regression?
- Can the organization prove that controls operated as designed?
Signal layers
Use linked signals rather than one oversized log:
- Platform: compute, accelerator, memory, storage, network, scheduling, and dependency health.
- Service: traffic, latency, errors, saturation, queueing, timeouts, retries, and availability.
- AI behavior: token or workload volume, retrieval support, citation, evaluation, refusal, and human-feedback indicators.
- Policy: authentication, authorization, route, inspection, approval, tool, and egress decisions.
- Lifecycle: artifact, release, deployment, configuration, rollback, and retirement events.
- Outcome: accepted completion, correction, escalation, abandonment, and business-relevant result where appropriate.
Protect telemetry
Prompts, retrieved passages, model responses, tool inputs, identities, and labels may be sensitive. Collect the minimum needed, prefer structured reason codes and references over raw content, separate access by purpose, set retention deliberately, and test deletion and export controls.
Correlation and integrity
Use stable correlation identifiers across services without treating them as authorization. Record synchronized time, component and policy versions, environment, and evidence integrity metadata. Protect the telemetry path from unauthorized alteration and monitor gaps.
Dashboards are views, not evidence by themselves. Retain the underlying versioned definitions, queries, alerts, incident decisions, and immutable release or audit records required to support a claim.
Module worked example
A release has passing functional tests but no evidence for a required access-control objective.
- Apply the control
- Link the missing objective to an owner and hold the release pending evidence or a valid scoped exception.
- Expected result
- The gate records the unresolved finding and the specific condition needed to reconsider it.
- Troubleshooting
- Check evidence scope, revision and expiry. A high average coverage score cannot satisfy a missing critical control.
Practice and completion check
Choose one control from this lesson. Identify its actor, target, enforcement point and expected result. Describe an allowed case and a denied or failed case. For a runnable exercise, retain observed output and separate it from expected output. Record unresolved limitations before marking the lesson complete.
Your lesson progress
Completion is self-reported learning progress; it does not certify practical competence.
Status: Not marked
Loading your learning record…