Tekoälyavusteinen testisuunnittelu ja LLM-arviointi asiakaspalveluagenteille
In short
Toimiala: Energia
Ratkaisu: Täysin uudenlainen asiakaspalvelumalli, jossa tekoäly toimii ensimmäisenä kontaktipisteenä ja osana asiakkaan palvelupolkua. Useita tekoälypohjaisia asiakaspalveluagentteja kehitettiin ja ylläpidettiin yhdessä kanavakohtaisten tietokantojen, testijoukkojen ja arviointikonfiguraatioiden kanssa.
Toteutustapa: LLM-pohjaisten agenttien laadunvarmistus, erityisesti vastausten oikeellisuuden, olennaisuuden ja hallusinaatioiden hallinnan automatisoitu arviointi hyödyntäen LLM-as-a-judge-mallia, jolloin testaus perustuu semanttiseen laatuun tarkan merkkijonovertailun sijaan.
Avainsanat: GitHub Copilot, DeepEval, LLM-as-a-judge, Regressiotestaus, Testiautomaatio
What was done in practice
The objective was to make AI agent testing repeatable, version-controlled, and scalable across business domains and channels. In practice, the team built a standardized test set and folder structure where question-answer pairs, evaluation settings, and reference data are maintained in version control. DeepEval automations were used to assess metrics such as correctness, answer relevancy, toxicity, and refusal behavior.
GitHub Copilot was also used to support test design and test preparation work, especially when drafting question-answer variations, shaping test case structures, and preparing JSON-based test set content from source material. This sped up the creation of consistent test assets.
Human validation was carried out based on the agentic evaluation results and findings.
Results
First, the amount of executed tests increased with test data automation compared to manual data analysis and test case writing. Automatic evaluation and pre-validation made results analysis far more efficient than having a human tester validate every result.
Customers get answers and can handle their matters immediately, with over 80% reporting a positive experience interacting with the AI. The solution has lightened the load on customer service and created a scalable foundation for continuous development.
