Observability and operational evidence
Design signals that explain service health, AI behavior, policy outcomes, change, and incidents without leaking sensitive content.
Before you start
Module prerequisites: SAI-100. Check the exercise-specific tools and environment before running commands.
Module outcomes: Design a private ingestion and retrieval flow; Implement permission-aware retrieval; Evaluate grounding, citations, and no-answer behavior. This lesson contributes to that sequence.
Observe the complete AI service
Infrastructure health alone cannot explain an AI system. Operators need correlated signals across request handling, identity and policy, retrieval, model inference, tool execution, user outcomes, change, and evidence production.
Design telemetry around questions the team must answer:
- Is the service available within its approved capacity and latency envelope?
- Are users receiving grounded, permitted, and useful outcomes?
- Which model, prompt, corpus, policy, tool, runtime, and configuration produced a result?
- Are denies, redactions, escalations, no-answer outcomes, or unsafe attempts changing?
- What changed before an incident or quality regression?
- Can the organization prove that controls operated as designed?
Signal layers
Use linked signals rather than one oversized log:
- Platform: compute, accelerator, memory, storage, network, scheduling, and dependency health.
- Service: traffic, latency, errors, saturation, queueing, timeouts, retries, and availability.
- AI behavior: token or workload volume, retrieval support, citation, evaluation, refusal, and human-feedback indicators.
- Policy: authentication, authorization, route, inspection, approval, tool, and egress decisions.
- Lifecycle: artifact, release, deployment, configuration, rollback, and retirement events.
- Outcome: accepted completion, correction, escalation, abandonment, and business-relevant result where appropriate.
Protect telemetry
Prompts, retrieved passages, model responses, tool inputs, identities, and labels may be sensitive. Collect the minimum needed, prefer structured reason codes and references over raw content, separate access by purpose, set retention deliberately, and test deletion and export controls.
Correlation and integrity
Use stable correlation identifiers across services without treating them as authorization. Record synchronized time, component and policy versions, environment, and evidence integrity metadata. Protect the telemetry path from unauthorized alteration and monitor gaps.
Dashboards are views, not evidence by themselves. Retain the underlying versioned definitions, queries, alerts, incident decisions, and immutable release or audit records required to support a claim.
Module worked example
Two users ask the same finance question, but only one may read the source document.
- Apply the control
- Filter active documents by the caller's permissions before retrieval context is assembled.
- Expected result
- The permitted actor gets a supported citation; the denied actor receives no finance context.
- Troubleshooting
- Inspect retrieved document IDs and cache identity. Removing a citation after generation cannot undo disclosure.
Practice and completion check
Choose one control from this lesson. Identify its actor, target, enforcement point and expected result. Describe an allowed case and a denied or failed case. For a runnable exercise, retain observed output and separate it from expected output. Record unresolved limitations before marking the lesson complete.
Your lesson progress
Completion is self-reported learning progress; it does not certify practical competence.
Status: Not marked
Loading your learning record…