# SovAIHub model selection scorecard

Revision: 2026-09-26. Editable planning template; no model is pre-approved.

## Decision context

- Workload and business owner:
- Data classification and permitted processing/storage locations:
- Inference, retrieval, embeddings, telemetry, backup, and support boundaries:
- Users, languages, input/output lengths, arrival rate, and active concurrency:
- Quality, security, p95 latency, availability, and cost acceptance criteria:
- Existing hardware and operating team:
- Decision date and approvers:

## Pass/fail gates (evaluate before scoring quality)

| Gate | Required evidence | Candidate A | Candidate B | Baseline |
| --- | --- | --- | --- | --- |
| Data/control boundary | Approved data-flow and access map | | | |
| Use and redistribution rights | Model/artifact license and terms review | | | |
| Runtime support | Exact model, format, accelerator, and feature support | | | |
| Security | Retrieval/tool authorization, retention, tenant isolation | | | |
| Operational ownership | Patching, incidents, capacity, recovery owners | | | |

A failed or unresolved mandatory gate prevents approval. A weighted quality score cannot override it.

## Pin each candidate

Record model repository, immutable revision/digest, tokenizer, chat template,
quantization artifact and configuration, serving image digest, driver versions,
decoding/reasoning settings, context/output limits, tool parser, and license.
Record provider version/deprecation behavior for managed endpoints.

## Compare on the same holdout set

| Measure | Definition / target | A | B | Baseline | Sample count / uncertainty |
| --- | --- | --- | --- | --- | --- |
| Task success | Workload-specific acceptance | | | | |
| Evidence and citations | Claim support + coverage, where applicable | | | | |
| Abstention | Missing-evidence success + unnecessary refusal rate | | | | |
| Security | Any unauthorized disclosure or action blocks | | | | |
| Latency | p50/p95 first-token and end-to-end; queue time | | | | |
| Goodput | Accepted work meeting service targets / unit time | | | | |
| Memory | Peak measured use on the target configuration | | | | |
| Unit economics | Period attributable cost / accepted outcomes | | | | |
| Operations | Support effort, update process, failure recovery | | | | |

Report per-language, task, role, and long-input slices. Include reasoning tokens,
retries, tools, evaluation, and human correction in the agreed cost boundary.
Keep unsuccessful requests in cost accounting. Mark zero-denominator measures
undefined or not applicable; do not label them perfect.

## Decision record

- Recommended candidate and evidence:
- Rejected alternatives and reasons:
- Approved fallback and rollback procedure:
- Unresolved risks, mitigations, and owners:
- Hardware/cost assumptions requiring measurement:
- Model, data, runtime, or demand changes triggering reevaluation:
- Reviewer decisions and evidence locations:

Guide: https://sovaihub.com/resources/model-selection-guide