AI evidence-pack workshop

Workshop objective

Build a traceable assurance package and release recommendation for a synthetic private knowledge assistant moving from pilot to controlled internal release, then run the executable LAB-250-01 evidence-pack assembler to see the same traceability, evidence-quality, and release-gate logic applied and tested automatically. Use synthetic policies, fictional identities, and placeholder findings only — do not submit real customer or production evidence into this exercise.

Content review

2026-08-09

Automated lab validation

2026-08-09

Independent reproduction

Not yet completed

Scenario

The assistant answers employee questions using SAI-230-style permission-aware retrieval, runs inside an approved private environment, and is now proposed for controlled internal release beyond its pilot group. Reuse the system definition, trust-boundary map, and control-objective register produced in SAI-100 and SAI-120; do not redefine the system from scratch.

Part 1: Design the assurance package by hand

Complete these steps before running the executable lab, so the automated release-gate output is checked against your own reasoning rather than taken on faith.

Step 1: Define scope and claims

Record the exact release scope, intended and prohibited uses, stakeholders, applicable policy or regulatory claims, and the boundaries of what this review will and will not assess.

Exit check: a reader can state what is being assured and what is explicitly out of scope.

Step 2: Build the traceability matrix

Using the downloaded template, map each risk or obligation to a control objective, implementation, test, evidence record, owner, reviewer, and review frequency.

Exit check: every control objective has an owner and at least one evidence producer; gaps are listed as findings, not left blank.

Step 3: Assess evidence quality

For the five highest-impact controls, check each evidence record for identity, scope, time, source, integrity, and retention. Flag any record that depends on an unversioned screenshot or dashboard view with no underlying machine-readable source.

Exit check: every retained evidence record can be tied to a specific system version.

Step 4: Assemble the system passport

Build a system passport that links to — rather than copies — the authoritative artifacts produced in Steps 1–3 and in earlier modules.

Exit check: a reviewer unfamiliar with the project can identify the release candidate, its boundary, its controls, and its known limitations from the passport alone.

Step 5: Evaluate release criteria

Evaluate the release against its acceptance criteria. Document failures and exclusions separately from a passing summary — do not let an aggregate result hide an individual failed criterion.

Exit check: every failed or excluded criterion is visible, not folded into an overall score.

Step 6: Draft exceptions and the recommendation

Draft exceptions for any unmet criteria, each with compensating controls, an owner, an approver, and an expiry date. Issue an approve, approve-with-conditions, hold, or reject recommendation for the exact candidate, naming the decision authority.

Exit check: every exception has an expiry; the recommendation names one specific candidate, not the product in general.

Step 7: Define monitoring and reassessment

Define the monitoring window, rollback triggers, and the material-change events (see Controlled change and versioning) that will force reassessment before this release's evidence can be relied on again.

Exit check: the rollback trigger list and the reassessment trigger list are not the same list.

Part 2: Run the executable evidence-pack lab

The executable source is maintained in academy/labs/LAB-250-01. It ships a synthetic control catalog, evidence records, and a system passport, and computes the same traceability matrix, evidence-quality findings, exception coverage, and release recommendation you built by hand in Part 1.

cd academy/labs/LAB-250-01
python scripts/validate.py

python -m app.cli

Release-gate outcomes

Evidence contract

Every assembled evidence report identifies the control-catalog, evidence-record, and system-passport versions it was computed from, alongside the traceability matrix, findings, covered findings, and expired exceptions.

{
  "lab_id": "LAB-250-01",
  "release_candidate": "private-knowledge-assistant-v1.4.0",
  "recommendation": "hold",
  "versions": {
    "control_catalog": "1.0.0",
    "evidence_records": "1.0.0",
    "system_passport": "private-knowledge-assistant-v1.4.0"
  },
  "findings": [
    {
      "control_id": "CTRL-INJECTION-01",
      "reason": "evidence missing required fields",
      "covered_by_exception": false
    }
  ],
  "covered_findings": [
    {
      "control_id": "CTRL-DELETION-01",
      "reason": "no evidence recorded",
      "covered_by_exception": true
    }
  ]
}

Participant extension

Add a control, an evidence record, or an exception to the synthetic dataset, or change evaluated_at in the system passport to expire the existing exception. Record the new data version and rerun every acceptance gate. A second operator must be able to follow your changes without relying on undocumented steps before the lab can move from prototype to reproducible.

Workshop package

Deliver:

1. Scope, claims, and exclusions.
2. Control-to-evidence traceability matrix.
3. Evidence-quality review.
4. System passport.
5. Findings and exceptions register.
6. Release recommendation naming the exact candidate.
7. Monitoring, rollback, and reassessment plan.
8. The LAB-250-01 validation report and evidence bundle.

Another reviewer should be able to reconstruct the decision without relying on unstated context. This workshop is an educational assurance design exercise, not an audit opinion or production release authorization.