What was done in practice

The primary objective for VALA was to establish a test automation foundation for future regression testing, focusing on both the AI agent and the UI components.

The AI-specific testing involved an AI agent analyzing specifications and code to generate a test plan. This plan was reviewed by an expert, and PyTest automation scripts were developed using the plan as a source document. AI was also used to generate test data by analyzing existing datasets and randomizing or anonymizing them for testing purposes.

For the UI, the Playwright planning agent analyzed specifications and code to generate a test plan. After expert review, TypeScript automation scripts were created using the Playwright generator agent, an iterative process as changes were frequent throughout the project. The Playwright healer agent, with subject matter expert supervision, was also used to detect and resolve broken tests automatically.

Finally, documentation including executive summaries was produced using an AI agent to analyze the resulting code and tests, and Finnish translations were drafted using an AI agent optimized for proper form and grammar in the Finnish language.

VALA was also responsible for developing extensive testing for the client’s personnel to perform. Acceptance tests were prepared for salespeople and admins, with test execution instructed and supervised, concluding with a summary report on the process, defect reporting, and handling. The project’s final retrospective acknowledged the significant value added by the testing procedures.

Results

The new TA Playwright framework, built from scratch in one month, executes API and UI tests across the live full stack in two environments, with tests redesigned as isolated, self-contained scenarios to resolve previous flakiness.

While formal metrics are pending, test coverage and delivery speed have improved together. AI accelerates every phase, from specs to code, providing a reliable 85–90% starting point that requires human refinement, hinging on high-quality documentation and project-specific conventions.

The work is still in progress, but the current state of the test suite across the project is as follows:

Time saving for end-users of the AI-based loan request proposal system is estimated at 20 minutes: the fully manual process took about 30 minutes, and the AI assisted process takes approximately 10 minutes, a 3x performance improvement.

AI and UI tests were planned, implemented, and ran with a passing verdict in the pipeline against the development environment setup, with implementation and testing finished within the scheduled time.

10 minutes saved per loan application → 400 hours more time for sales in a year.