Once a model is stable and producing acceptable outputs in normal conditions, the harder question begins: how does it behave when things go wrong, either accidentally or maliciously? This phase combines security testing, ethical evaluation, runtime protection, and regulatory readiness.

1. Red Teaming and Adversarial Testing

Testing the “happy path” is only the beginning. Mature AI quality programs run dedicated red team exercises – structured attempts to make the system fail. For example:

• Jailbreak attempts
• Data leakage
• Privilege escalation via tools
• Unsafe tool execution
• Excessive agency
• Malformed or ambiguous inputs
• Attempts to bypass business rules

For each category, document attack scenarios, run them as eval cases, and verify that defenses hold. Red teaming findings should also become regression tests.

2. Prompt Injection: Direct and Indirect

Direct prompt injection is the obvious case: a user pastes “ignore all previous instructions” into a chat interface. The harder variant is indirect prompt injection, where the model reads hostile instructions from an untrusted data source like a web page or email. 

Protecting measures:

• Treat retrieved/external content as untrusted data
• Separate instructions from data
• Restrict tool permissions
• Require authorization outside the model
• Test whether injected content can change system behaviour
• Verify that sensitive context cannot be exfiltrated

3. Runtime Guardrails

Guardrails enforce defenses in production. The mature pattern is a layered runtime check around both inputs and outputs: input filtering, output filtering, tool authorization, and PII detection. Guardrails themselves must be tested too.

4. Compliance and Governance

A robust AI quality program aligns with major frameworks like the EU AI Act, NIST AI RMF, and ISO/IEC 42001. Quality assurance provides evidence of compliance through transparency, traceability, and defined error rates. 

AI compliance testing is not a separate test phase. Many of the required controls depend on the same traceability, monitoring, risk testing, and documentation that mature QA already produces.

5. Shadow Testing and Continuous Observability

Evaluation continues in the live environment. Continuous observability catches what offline evaluation misses: novel user inputs, real-world distribution shifts, and emerging failure patterns. Use shadow testing to compare candidate versions against production traffic without affecting users.

The next blog in our EvalOps series covers output intelligence and the management of stochastic results.

• Blog
|
• AI system QA
|
• EvalOps