AI-assisted test design and LLM evaluation for customer service agents
In short
Industry: Energy
Solution: A completely new customer service model where artificial intelligence acts as the first point of contact and as part of the customer’s service path. Multiple AI customer service agents were developed and maintained, along with channel-specific knowledge bases, test sets, and evaluation configurations.
Type of work: QA for LLM-based agents, especially automated evaluation of answer correctness, relevancy, and hallucination control, using an LLM-as-a-judge setup so testing is based on semantic quality instead of strict string matching.
Keywords: GitHub Copilot, DeepEval, LLM-as-a-judge, Regression testing, Test automation
What was done in practice
The objective was to make AI agent testing repeatable, version-controlled, and scalable across business domains and channels. In practice, the team built a standardized test set and folder structure where question-answer pairs, evaluation settings, and reference data are maintained in version control. DeepEval automations were used to assess metrics such as correctness, answer relevancy, toxicity, and refusal behavior.
GitHub Copilot was also used to support test design and test preparation work, especially when drafting question-answer variations, shaping test case structures, and preparing JSON-based test set content from source material. This sped up the creation of consistent test assets.
Human validation was carried out based on the agentic evaluation results and findings.
Results
First, the amount of executed tests increased with test data automation compared to manual data analysis and test case writing. Automatic evaluation and pre-validation made results analysis far more efficient than having a human tester validate every result.
Customers get answers and can handle their matters immediately, with over 80% reporting a positive experience interacting with the AI. The solution has lightened the load on customer service and created a scalable foundation for continuous development.
