Runtime and endpoint architectures

Separate responsibilities

Describe model storage and loading, inference runtime, accelerator scheduling, API translation, authentication, authorization, rate and quota enforcement, routing, safety or policy inspection, telemetry, and deployment management as distinct responsibilities. A single product may combine several of them, but the architecture documentation should not hide who is actually responsible for each one — when something fails, "the serving stack" is not an owner.

Endpoint contract

Define accepted request shapes, size and context limits, streaming behavior, timeout and cancellation semantics, response structure, error and refusal codes, version negotiation, identity context, idempotency where relevant, and evidence correlation. Treat this contract as something clients depend on independently of the runtime behind it.

Avoid exposing runtime-specific features through a supposedly portable contract unless the dependency is intentional and explicitly recorded — a client that silently depends on one runtime's quirks becomes an unplanned migration blocker later, documented in Architecture decision records.

Isolation and tenancy

Decide whether models, accelerator memory, caches, queues, logs, and administrative controls are shared across workloads or tenants. Document tenant and data separation, scheduling fairness, noisy-neighbor behavior, model-loading authority, and cleanup between requests. An isolation decision made implicitly by whatever the runtime defaults to is still a decision — it should be reviewed, not inherited by accident.

Failure paths

Specify what happens when a model cannot load, context is too large, the queue is full, a stream is interrupted, policy is unavailable, or evidence cannot be recorded. Clients need stable, non-misleading failure signals rather than a generic error that could mean anything from "retry immediately" to "this request will never succeed."

Retrying an unsafe or non-idempotent operation can amplify harm — a client-side retry policy designed for a typical web API can be actively dangerous against a model endpoint that performed a partial or side-effecting action before failing. Define which failure classes are safe to retry automatically and which require a fresh, explicitly authorized request.