The quality of an AI application cannot rest on the model alone. Regardless of how smart the model is, the surrounding pipeline, interfaces, and infrastructure must function reliably. While AI itself operates stochastically, its technical backbone should behave as predictably and consistently as possible. completely deterministic. This is the first and most foundational phase of quality assurance: preventing the costliest and most easily avoided errors.

We call this System Robustness.

1. Technical Foundation

Quality assurance begins with ensuring that data moves correctly. We verify that API calls, authentication, and connections are sound, that integrations work as intended, that data travels in the correct format to the correct destination, and that access control prevents unauthorized use.

For LLM-powered systems this also means handling provider failures gracefully. LLM APIs introduce failure modes that require explicit handling; rate limits, timeouts, partial responses, and provider-side incidents are routine. The pipeline must implement Retries, Circuit breakers, and Fallback strategies.

• Exponential backoff and retry strategies
• Idempotency keys to make retries safe
• Circuit breakers when a provider is unavailable
• Fallback strategies, such as a smaller model, cached response, or graceful degradation

Without a rock-solid foundation, the model’s intelligence is irrelevant.

2. Schema Control and Structured Outputs

Generative AI is prone to inventing non-existent structures or returning incorrect data types. The pipeline must include automatic safeguards that ensure generated JSON or other structured outputs comply with agreed-upon schemas.

Modern model APIs offer features that make this much easier:

• Structured outputs: the provider enforces a JSON schema at the decoding level
• Function calling and tool use: typed parameters with validation
• JSON mode: guarantees valid JSON but not schema conformance

Even with these features, server-side validation is mandatory. Schema discipline prevents downstream system crashes when the model occasionally produces something unexpected – as it will.

3. Prompt Versioning and Management

Prompts should be managed with the same discipline as code. They control the behavior of the model just as application logic controls the behavior of a service. Yet prompts are often kept in scattered text files, hard-coded strings, or worse, only inside an engineer’s head.

A mature pipeline treats prompts as first-class artifacts:

• Version control: every prompt change is reviewed and traceable
• Environment-specific deployment: prompts can be promoted from dev to staging to production
• A/B testing infrastructure: multiple prompt versions can be served simultaneously
• Regression testing: changes to prompts, models, or context configuration trigger the same eval suite as changes to code.

Without this discipline, a “small wording fix” can silently degrade quality across thousands of users.

4. Resource Stability

AI introduces a category of financial risk that traditional QA has not had to address: per-request cost variance. One poorly crafted prompt – or one user who exploits an open-ended interface – can cause API costs to spike by orders of magnitude.

Test response times and budget constraints in every build:

• Set hard token limits per request and per session
• Monitor input vs output token ratios (output tokens typically cost 3–5× input)
• Watch for agentic loops where the model calls itself repeatedly
• Define cost SLOs and alert on breaches

Token consumption should be predictable, measurable, and capped.

5. Regression Testing with a Golden Test Set

Maintain a curated set of representative inputs – a golden test set – and run it on every meaningful change to detect unexpected changes in response structure, latency, core behavior, or evaluation scores.

Because model outputs remain stochastic even with tightly controlled configurations, regression testing should focus on defined thresholds and expected behavior rather than exact output matching. If a regression occurs, the system should alert immediately.

6. Environment Parity

A common source of production incidents is environment drift: dev runs on one model version, staging on another, production on a third. Or the prompts are different. Or the embedding model has been updated in one environment but not others.

Treat environment parity as a hard requirement:

• Same model version across environments unless deliberately testing
• Same embeddings, vector indices, and retrieval configuration
• Same prompt templates and tool definitions
• Same guardrails configuration

When something works in staging but breaks in production, it is almost always because of parity drift.

This was the first post in our EvalOps blog series. The next one covers defensive operations and validation under adversarial conditions.

• Blog
|
• EvalOps