Test automation is not a magic button that can solve all your problems when pushed. Test automation is only a tool. A very powerful tool if used correctly. Still, if used in a wrong manner, it can do more damage than good. When thinking about implementing test automation, the first things to keep in mind are the pros and cons of it. Especially the cons.
Test automation requires longer initial development time and it is not cheap – this is one of the most important things to remember when deciding if/when to start a test automation process. Even if all other things required are in place (application can be automated, enough skilled people etc.), test automation still takes significant time to produce results. The initial higher development cost and slowness is something to keep in mind. Test automation only shows benefits in mid long term. If a test case is properly automated, the benefits of having that test case automated will only be clearly visible, once the test case is executed in many regression cycles. The cost of testing the feature covered by that automated test decreases over time. Compared to the manual execution, the cost stays the same no matter how many times the test is executed.
Five things to evaluate before test automation
1. Planning and strategy
Well-functioning test automation requires good test planning and good regression strategy. As mentioned, test automation is only a tool. Deciding what to automate, needs human input and not everything should be automated. You should only automate scenarios that provide enough added value. Automating something takes time, it might not be trivial, so it has to be balanced by the benefits that the automated test case brings. Good candidates for automation can be test cases which:
– take a lot of time to execute
– have lots of data validation
– are executed in each regression cycle
– are boring/difficult to be executed manually
– are business critical
Also, in some cases, test automation is the only way that a scenario can be executed. Examples of tests that are usually done only with test automation are:
– non-functional testing
– stability testing
– performance testing
– benchmarking testing
– stress testing
2. Right people
Automation requires skilled engineers. One of the most important factors that implicates whether a test automation process is successful or not, is the skill of the people involved in the effort. One should not expect to achieve good test automation without developer support. There are tools/sales people that advertise magic solution that require no development skills and promise great results. This has nothing to do with reality. Don’t trust these tools/sales people, they either have no idea what they are talking about or they are trying to take advantage of your trust.
Test automation is closer to development than it is to testing. The test cases and framework are actually production code. Writing/maintaining that code should be done following development rules. In some cases automating a certain scenario is not trivial and requires quite advanced development skills in order to add the support to the test automation framework for performing such actions – again, you need developers to be able to do that. There are now good frameworks available that allow better differentiation between developers and test engineers – this way test automation tasks can be better divided to suite everyone’s skills. Engineers that have more development skills can focus on developing the framework and the engineers that have more product knowledge/testing skills can use the framework/tools created by their colleagues to automate the manual test cases.
Support from developers and other important instances is the foundation for successful test automation. During development we should already be thinking about the most important question (from an automation perspective): How do we make the code testable?
Collaboration with the teams that are developing the software under test might seem like a no-brainer. Unfortunately, the fact is that quite often the support is not provided or it is not comprehensive enough. Some applications might not have been built with test automation in mind. In these cases, small changes in the application to facilitate easier automation, can help. For example, in GUI testing ensuring that there are unique identificators for certain GUI elements will help test automation a lot. If the developers take part in writing the tests (at least fixing the tests when changes are made), it naturally leads into a maintainable and high quality test code.
4. What do you need to test?
It is important to understand your software and its testing requirements before you head into test automation. You should start the process by evaluating what kind of testing is needed. Here’s a short list of what would be wise to test manually and what automatically:
Cases when manual testing is usually more suitable:
– Exploratory testing
– Usability/ look-and-feel testing
– New application/functionality (manual tests are always the first step in testing)
– Very complex functionalities
Cases when automated tests are usually more suitable:
– Regression testing
– Repetitive (boring) tests
– Performance testing
– Smoke testing
– Data driven testing
So, when you evaluate your situation, try to think what is the situation in hand and what kind of tests you need to do. If you have a brand-new application, you should start with well thought out and documented manual tests. When your environment is more mature you can move on into automating those same test cases. Keep in mind that manual testing should always (well, for now) have a part in your quality assurance process. On the other hand, there are some aspects of testing that can’t be executed manually, like performance and load testing.
5. Evaluating expected results and comparing them with the investment
We realise that it’s extremely hard to evaluate the results of your future test automation, especially if you’ve never initiated it before. However, you should do your best to get some idea of what kind of results would be possible to achieve in your situation. You should at least think of these dimensions:
– Increase in test coverage
– Increase in test accuracy
– Saved working hours when test automation is in a mature phase
When you try to evaluate the expected results to the size of the investment you can use this checklist of the things you will most probably achieve with test automation. Try to think about your case and how beneficial these achievements are to you.
With successful test automation, you will most probably achieve:
– Overnight testing
– Automated testing reports
– Increased test coverage in same or shorter time
– Increased repeatability for your tests
– Faster test coverage
If you can make educated estimations of possible results and these seem more valuable than the investment, then you have a good change that test automation could be suitable for you. If also the other aspects discussed in this text are under control, we suggest you stop thinking and start doing. We’d love to help you getting started, contact us here.
There you go. We hope the information provided helped you with your questions about test automation. Next you might like to read some of the following posts:
Images from the top:
- Aziz Acharki, unsplash.com
- Igor Ovsyannykov, unsplash.com