Practical test kit / Quality & release readiness
Find unsupported answers, retrieval failures, and permission leaks before they reach users. Establish a repeatable evaluation of evidence, citations, abstention, and security across every RAG release.
Reviewed · Vendor-neutral guidance
Use this checklist before running the protocol. Completion measures preparation only; it does not score your RAG system or certify release readiness. Selections reset when you reload.
0 / 10 prepared
01 / Decision framework
A fluent answer can be wrong, unsupported, stale, or visible to the wrong person. Keep these failure modes separate so the fix reaches the right component.
| Test slice | Expected behavior | Evidence to inspect |
|---|---|---|
| Answerable questions | Answer the question using sufficient authorized evidence. | Gold evidence IDs, retrieved chunks, final claims, and citations. |
| Missing evidence | State the limitation or request clarification without inventing a fact. | Abstention on unanswerable cases and unnecessary refusal on answerable cases. |
| Conflicting / stale sources | Apply an explicit source-authority and effective-date policy, or surface unresolved conflict. | Document versions, supersession metadata, and the chosen source. |
| Citation integrity | Cite passages that actually support the associated claim. | Both citation correctness and coverage of claims that require evidence. |
| Cross-role / cross-tenant access | Exclude unauthorized material before it reaches generation. | Identity, retrieval authorization decisions, cache isolation, response, and traces. |
| Document injection / tool misuse | Treat document instructions as untrusted content; reject unauthorized actions. | Output, attempted tool calls, arguments, network destinations, and permission decisions. |
Access control belongs in retrieval and tool enforcement. A prompt asking the model to respect permissions is not an authorization mechanism. Test warm caches and multi-turn sessions as well as fresh requests.
02 / Decision framework
Publish sample counts and results by language, document type, role, and difficulty. An overall average can hide a severe failure in a small but important slice.
For each answerable case, measure recall of labeled relevant evidence at the chosen k. Report claim support as supported factual claims divided by evaluated factual claims. A faithful answer can still be incomplete or answer the wrong question.
Citation correctness = supporting citations / citations checked. Coverage = evidence-requiring claims with valid support / evidence-requiring claims. Track task correctness separately and mark empty denominators as not applicable, never automatically perfect.
Report correct abstentions / unanswerable cases and unnecessary abstentions / answerable cases. Record any unauthorized retrieval, disclosure, or tool action as a separate critical failure; never average it away with quality scores.
Calibrate automated judges against a human-labeled sample, review disagreements, and pin the rubric and judge version. Repeat variable cases. Record latency, timeouts, retries, and cost per accepted answer alongside quality.
03 / Decision framework
The download contains eight small synthetic cases with source text and expected behavior. They are a starting contract for manual review or an adapter to your evaluator, not a statistically representative benchmark.
Read the protocol and map the fixture identities, permissions, and source IDs to a disposable test corpus. Keep synthetic fixtures separate from production documents.
Add approved real-world cases and split development from holdout evaluation. Freeze corpus, model, prompt, chunking, embedding, reranker, and policy versions.
Capture actual retrieved content and application responses for each identity. Review evidence and outcomes with the supplied rubric; do not send restricted traces to an external judge without approval.
Block critical security failures, missing required results, and unresolved evaluator errors. Agree quality floors and permitted regression per slice before testing, then retain failed cases for the next release.
Passing a finite suite does not prove zero hallucination. Combine evaluation with production sampling, incident triage, source maintenance, and a tested rollback path.
Working materials
Download the editable materials, adapt them to your environment, and keep the assumptions alongside the results.
Work with SovAIHub
Scope an evaluation engagement around your corpus, identities, and most consequential failure modes.
Bring a representative question set, source lifecycle rules, user roles, the RAG interface, and your definition of an unacceptable answer. Access to sensitive data can remain inside your boundary.
Scope, acceptance criteria, delivery schedule, and fees are agreed before implementation.
Scope a RAG evaluationConnect the decision to the next engineering or operating step.
Official documentation checked for this revision. Confirm the exact model, runtime, and license versions when implementing.