Agentic test automation development and test data generation with LLM evaluation for loan application AI
In short
We developed regression test automation for a loan request proposal process, utilizing AI to analyze specifications and the codebase. With the assistance of AI, anonymized test data was generated. The testing strategy also incorporated LLM evaluation of the output produced by the AI agent. Furthermore, AI agents were employed to analyze UI specifications and the codebase to generate comprehensive test scenarios and cases.
AI and UI tests were planned, implemented and ran with passing verdict in pipeline against development environment setup. The project implementation and testing was finished within the scheduled time.
Client: Suomen Hypoteekkiyhdistys
Industry: Finance
Product: AI-based document processor for pre-screening incoming loan proposals and a UI for loan request handlers.
Type of work: Test automation utilizing AI to analyze specifications and the codebase. LLM evaluation of the output produced by the AI agent. AI agents analyzing UI specifications and the codebase to generate comprehensive test scenarios and cases.
Tools: GitHub Copilot with VS Code, Playwright

What was done in practice
The primary objective for VALA was to establish a test automation foundation for future regression testing, focusing on both the AI agent and the UI components.
The AI-specific testing involved an AI agent analyzing specifications and code to generate a test plan. This plan was subsequently reviewed by an expert, and PyTest automation scripts were developed using the plan as a source document. Additionally, AI was used to generate test data by analyzing existing datasets and randomizing or anonymizing them for testing purposes.
For the UI, the Playwright planning agent analyzed specifications and code to generate a test plan. After expert review, TypeScript automation scripts were created using the Playwright generator agent. This was an iterative process, as changes were frequent throughout the project. The Playwright healer agent with subject matter expert supervision was also utilized to detect and resolve broken tests automatically.
Finally, documentation including executive summaries was produced using an AI agent to analyze the resulting code and tests. Finnish translations were drafted using an AI agent optimized for proper form and grammar in the Finnish language.
The project was done in co-operation with Suomen Hypoteekkiyhdistys, VALA Group and Siili Solutions team members.
Acceptance testing
Our responsibility was also to develop extensive testing for the client’s personnel to perform. Acceptance tests were prepared for salespeople and admins, and test execution was instructed and supervised. The testing concluded with a summary report on the acceptance testing process, defect reporting, and handling. The project’s final retrospective acknowledged the significant value added by the testing procedures.
Why AI in this project?
Tight schedule constraints
AI was integrated into the development of test automation to address tight schedule constraints, with approximately one month of development before acceptance testing and release. The technology enabled the rapid drafting of plans and test scripts, allowing the test automation expert to focus on aligning tests with the actual implementation and documenting the code for future maintenance and development.
Help with test data generation
AI was used in test data generation and combined with human subject matter expertise to ensure the test set covered also all edge cases.
Results
The performance improvement due to time saving is estimated to be 33%. AI and UI tests were planned, implemented and ran with passing verdict in pipeline against development environment setup. The project implementation and testing was finished within the scheduled time.
33%
Performance improvement
97%
Test coverage for AI testing
81%
Test coverage for UI testing
100%
Functional testing coverage
208
Test sets
AI amplifies existing quality
Even though this was an AI-based development project, the testing part was based on proven quality assurance principles and procedures. AI during development was used as an assistant to help speed up development of test plans, test scripts and documentation. The test automation expert decided on direction and reviewed the generated material before it was taken into use.
Once again this project showed us the importance of good software quality practices and standards. AI amplifies existing quality. It does not create quality where none exists. Luckily, at Hypoteekkiyhdistys, the foundations of quality were at place.
Quote from the client
“AI added value when combined with clear testing practices and expert review. Human expertise remained essential throughout the work.”
Lauri Liukkonen, Product & Development Team Lead, Suomen Hypoteekkiyhdistys
Quote from VALA
“It was easy to incorporate AI as one of tools to do the groundwork of producing the bulk material. I was free to analyze and fine tune the actual beef in test plans and test code. The greatest time savings came from generating and translating the test documentation.”
Harri Törmänen, Senior test automation developer, VALA



