Observability and operational evidence
Design signals that explain service health, AI behavior, policy outcomes, change, and incidents without leaking sensitive content.
Before you start
Module prerequisites: SAI-100. Check the exercise-specific tools and environment before running commands.
Module outcomes: Define useful AI service indicators; Connect evaluation signals to operations; Measure capacity, unit cost, and accepted outcomes. This lesson contributes to that sequence.
Observe the complete AI service
Infrastructure health alone cannot explain an AI system. Operators need correlated signals across request handling, identity and policy, retrieval, model inference, tool execution, user outcomes, change, and evidence production.
Design telemetry around questions the team must answer:
- Is the service available within its approved capacity and latency envelope?
- Are users receiving grounded, permitted, and useful outcomes?
- Which model, prompt, corpus, policy, tool, runtime, and configuration produced a result?
- Are denies, redactions, escalations, no-answer outcomes, or unsafe attempts changing?
- What changed before an incident or quality regression?
- Can the organization prove that controls operated as designed?
Signal layers
Use linked signals rather than one oversized log:
- Platform: compute, accelerator, memory, storage, network, scheduling, and dependency health.
- Service: traffic, latency, errors, saturation, queueing, timeouts, retries, and availability.
- AI behavior: token or workload volume, retrieval support, citation, evaluation, refusal, and human-feedback indicators.
- Policy: authentication, authorization, route, inspection, approval, tool, and egress decisions.
- Lifecycle: artifact, release, deployment, configuration, rollback, and retirement events.
- Outcome: accepted completion, correction, escalation, abandonment, and business-relevant result where appropriate.
Protect telemetry
Prompts, retrieved passages, model responses, tool inputs, identities, and labels may be sensitive. Collect the minimum needed, prefer structured reason codes and references over raw content, separate access by purpose, set retention deliberately, and test deletion and export controls.
Correlation and integrity
Use stable correlation identifiers across services without treating them as authorization. Record synchronized time, component and policy versions, environment, and evidence integrity metadata. Protect the telemetry path from unauthorized alteration and monitor gaps.
Dashboards are views, not evidence by themselves. Retain the underlying versioned definitions, queries, alerts, incident decisions, and immutable release or audit records required to support a claim.
Module worked example
A service reports low token cost while many requests miss their latency target.
- Apply the control
- Calculate cost per accepted outcome and inspect errors, queue time and tail latency under the same load.
- Expected result
- The operating scorecard connects cost and capacity decisions to useful service outcomes.
- Troubleshooting
- Include timed-out and rejected work in the denominator rather than reporting only successful responses.
Practice and completion check
Choose one control from this lesson. Identify its actor, target, enforcement point and expected result. Describe an allowed case and a denied or failed case. For a runnable exercise, retain observed output and separate it from expected output. Record unresolved limitations before marking the lesson complete.
Your lesson progress
Completion is self-reported learning progress; it does not certify practical competence.
Status: Not marked
Loading your learning record…