Your Ultimate Guide to Test Automation: What, Why, What For and When to Apply

Test automation is one of the major novelty approaches that are here to stay. That’s because it’s not just a trend or a fad. It IS actually a good strategy to reduce manual labor, involve stable processes and, thus, foster testing and accelerate feedback, increase ROI on QA and decrease expenses.

According to the European Software Testing Benchmark report 2016, the majority of clients who made the best of test automation experienced a return on their test automation investment immediately (24%) or within the first 6 months (24%).

This article will help you understand what automated testing is about, its major peculiarities, pros and cons. Thus, for you this notion will transform from something intimidating and difficult to apply into a useful tool to invest in to boost your project’s success.

What Test Automation Stands For

To start with, here’s a short profile to help you grasp the essence of the notion itself.

What?

In a nutshell, test automation is a software testing procedure aimed at testing and comparing the actual results with the expected ones.

How?

It is implemented by writing test scripts and automating them via test automation tools

Why?

Automation (when executed properly) enhances testing (and QA in general) and makes it:

• more systematic,
• more effective,
• more efficient,
• faster
• human error - free.

What for?

• to reduce the number of manual tests (if it is viable, cost-efficient and effective)
• to perform tests that are challenging to perform manually
• to save time, money and effort

Note!

1. The aim of test automation is NOT to replace manual tests or testers. It is to aid in cases when manual testing cannot or should not be carried out.
2. Automation should not be included by default into your SDLC (Software development life cycle). It is up to the QA team to assess every single project individually and decide on their reasonable use.

What You Should Know About the Testing pyramid

Our previous article focusing on the importance of QA proves that to develop a successful app, you need testing. Certainly, not just any testing, but good, thorough, well-planned testing. And that is what a test strategy should be specified for. In case of test automation, a respectable QA team is sure to use the test automation pyramid before shaping any strategy.

What?

The testing pyramid is a concept of a framework that strategically classifies software tests into major categories.

Why?

The pyramid defines

• what test levels should or should not be included into an automated test suite
• the sequence and frequency of automated tests.

What for?

It helps the QA and development teams to:

• reduce time and efforts spent on finding bugs and their causes
• ensure high quality testing
• build reliable test suites.

The major focus is on timely feedback to make sure existing features are not disrupted by any code changes.

The main layers of the pyramid are as follows:

1. UI tests
2. Service tests (integration and API)
3. Unit tests

Let’s discuss each level in detail, starting from the bottom up.

Unit tests make the solid basis of the test automation strategy.

Pros:

• composed in the same programming language as the app itself, thus, often generated by developers
• fast to create, provide feedback to developers to help them make changes.
• test one variable to make sure it works as expected under isolated conditions.
• involve virtual devices which are faster and cheaper to use.

Cons:

• have a limited scope as they cover individual units or functionalities with no relation to external factors.
• require a great number of scenarios

Service tests help verify core logic without recourse to the UI.

The two types of tests on this level are:

A. Integration Testing - performed to check how different codes, features, units interact with each other i.e. how each part of the code communicates with the external components.

B. API testing - performed by sending calls to the API and obtaining output to verify the response of the whole system to the input parameters.

All of these types are executed on the second level of the pyramid to make sure the development runs error-free. They also have their advantages and disadvantages.

Pros:

• help to test the whole application early without a UI
• run less frequently than Unit tests

Cons:

• require pre-production environment to be run in.

UI tests are the most difficult to automate as they reflect the user’s perspective and test how the app works from start to finish.

Pros:

• are carried out more rarely
• involve both emulators and real devices to mimic the actions on users’ gadgets
• implement user simulation (conflicting apps, network latency, call interruptions etc.) to test real-life conditions

Cons:

• cannot be executed until the development and design are complete
• take longer to complete
• are complicated and fragile as they involve more dependent factors and multiple user scenarios

Now, when you are aware of what all the levels stand for, it’s time to draw the following conclusion. The Test Automation Pyramid shows that:

1. The higher you go, the more costs it takes to create, maintain, execute and cover the tests;
2. The lower you start, the more automated, cheaper and faster it is.

Tip 1. The most reasonable way for beginners is to push automated tests as low as possible and get a higher ROI.

Why? Unit tests are small and isolated so detecting bugs becomes faster and cheaper to execute. Tests on the top of the pyramid are time consuming, sometimes even brittle, fewer and more often tend to be carried out manually.

Tip 2. Sticking to test automation on one level to avoid additional costs is not an option.

Why? To perform successful automated testing every layer of the pyramid should be analyzed. Choosing what and how many tests to automate and what to remain manual is a different issue. Yet, covering all of the levels is obligatory if you wish to get a high quality product.

Tip 3. The test automation pyramid should not be confused with the strategy itself.

Why? Banal application of this pyramid is just a framework rather than a universal solution. Any and every project should be discussed with your QA team, risks assessed and test automation strategy carefully selected and drawn up.

Our experience proves that the test automation pyramid can be really helpful for agile teams as it enhances speed and effectiveness. The matter is if you are lucky to have a good QA team, they are sure to introduce its logic into your test automation processes to manage their time more efficiently, run easier tests from the very beginning and, thus, save everyone’s time and money.

As a customer, you can also check if the QA team has the correct priorities. If they offer writing test scenarios for UI levels, you have every chance to doubt that their business logic and back-end functionality are correct. If they insist on that option, you should know it leads to more work, lower test coverage and lower quality of your project. Thus, by knowing how the pyramid must be implemented, you are sure to avoid such misunderstandings and choose truly skilled QA teams.

What Test Cases To Automate and What Tools To Choose

Once you understand the test classification and their features, we’ve come to one of the most relevant issues in test automation - selecting tests to be automated. The general rule is:

It is necessary and advisable to automate those processes which both the team and your business will benefit from rather than experience any discomfort or new challenges.

It happens that the QA teams are eager to automate any and every test they can, but such decisions eventually backfire and cause a number of problems with quality, performance and budget numbers. Here’s a cheat sheet for you to be sure if your QA team is on the right track.

Test cases should be automated when:

• The task will be repeated (numerously) without changes. Thus, running multiple similar scenarios simultaneously by automated testing is a game-changer.
• It is a large-scale project. In such cases, fixing bugs at the initial stages is crucial to the success of the whole mission and timely deployment.
• Automation saves time and costs (but doesn’t affect quality). Saving time is sure to enhance your team’s productivity and motivation while saving costs is sure to please you as the project owner.
• The test and/or the task and/or the requirements are stable. The risk of alterations should be as minimal as possible if not absent at all.

To help you set standards and priorities for test automation and weigh your chances of higher ROI on investing in it, here are some things to consider as well:

1. Top priority in test automation goes to unit testing while service tests and UI tests are next in line. We would definitely advise to focus on automating the lower two levels as it is faster, easier, and cheaper to accomplish not to mention they are highly reusable and reliable.
2. To make sure test automation is solid enough, bother to apply data from marketing research. Here critical success factors are if your team is aware of how your customers interact with similar applications, what common flows there may be, what user experience can lead to failure and what user personas can be designed to create network profiles to suit them the most.

Another important factor is tool selection. As it usually happens with highly-technological issues, there is no clear answer or a cheat sheet that gives precise instructions on what and how to use. Yes, the banal phrase works here perfectly - it always depends. Every project is special and, therefore, requires individual approach and thorough analysis. At LinkUp Studio we use the following guidelines to verify our choice of tools:

• What issues will the QA team use the tool for?
• Will it fulfill the requirements or solve problems we really have?
• Will it optimize our work?
• Is it reusable?
• Does it match our QA team’s skill set and tech stack?
• Does it fit in with the client’s budget?
• Are the reports provided by this tool detailed enough?
• Is it suitable for collaboration with other members of the project team, management or clients?
• Can this tool integrate with CI\CD tools?

Finally, it is always up to a QA team to make thorough analysis of all the factors involved to make an informed and well-grounded decision.

After all, applying tools is always a good idea once it helps save costs and time rather than add pressure or disparity. Therefore, the key step in either tool selection or test automation in general is choosing the right QA team who can add value to your project and assess the processes with expertise.

How You Can Benefit From Test Automation

So far, it must be obvious that the main aim of test automation is to make the QA process faster and with less effort involved. Yet, there are both benefits and downsides to every enhancement strategy. What you already get from this publication is that NOT everything CAN or SHOULD be automated. So, to get a better perspective of the issue, here’s what you should know about test automation.

Benefits of test automation:

• increased code, functionality and non-functionality test coverage 24/7
• faster feedback and outcomes available to the whole team in a unified manner
• better ROI
• reusability
• saves time and money once the test suites are established
• critical to regression testing, stress testing, reliability testing, load testing, and performance testing
• once properly automated, require little or no supervision or intervention
• generating accurate test data
• allows the team to focus on creativity rather than routine checking.

Yet, there are a few things you should keep in mind when engaging test automation:

• when making any deadline assumptions, always spare time (and funds) for effective(=beneficial) framework and infrastructure building, data collection and tools application
• never underestimate further maintenance once the project is deployed
• keep away from complete test automation on UI levels as it cannot be thorough enough

And finally, as it happens with any other process in project implementation, to avoid faulty or random results, opt for an experienced and scrupulous QA team that has both special expertise and skills.

To sum up, test automation certainly is a win-win as it increases software quality while making debugging and time to market faster. Talking numbers. Finding bugs at the early stage can save you 40x the cost of detecting it during the coding phase.

Yet, you realize that 100% is not just impossible but unnecessary. So, to assess this option objectively, you have to consider the other one, i.e. to understand the benefits and drawbacks of manual testing.

pros

cons

• provides a human=user perspective
• enable exploratory testing
• when combined with automated testing, mitigate the risk of faulty results of automated testing with insufficient scripts
• is the only type of testing capable of detecting very complex and deep-rooted errors that simply cannot be covered by automated tests

• requires more time and effort
• introduces human error

Your main takeaway at this point is the idea that testing automation does not replace testing in general. Automated tests lack domain knowledge and human intelligence, thus, they should never eliminate manual ones but complement them and reduce their number when it’s appropriate.

Good quality and successful delivery of your dream project are achieved by proper combination of both. The latter is possible through engaging a skilled QA team capable of finding the right balance between the two types of testing to ensure maximum efficiency in terms of technological features and user appeal.

How We Perform Test Automation at LinkUp Studio

The core method our QA team applies is CI\CD as it helps us to introduce test automation into various stages of the project development.

In this approach CI stands for continuous integration, while CD means continuous delivery and continuous deployment. The main reason we stick to this principle is that it helps to ensure flawless integration of new codes on every stage of alterations (CI) and error-free delivery and deployment in the long term (CD). Moreover, we are sure it provides ongoing monitoring and automation of the SDLC.

The core idea is to merge all changes, efforts, workers and ideas to enable the development process to run as a whole.

In order to make sure the clients see the outcomes and improvements introduced we are always ready to present them with thorough reports on any of the milestones of the QA process. Here are some samples from our previous projects for you to understand what a client can review and discuss.

Another LinkUp Studio project includes the Allure report containing detailed information about the results of conducting UI testing.

To sum up, we all know the better the QA, the better the deployment. And even if it seems outrageous or impossible to expect any step in development or QA to both save time\costs and enhance quality, test automation does appear to be the real deal. Certainly, as long as it is properly applied and well-planned. Like any other implementation of new software, tools or principles, test automation requires patience, accuracy and expertise to make you experience its benefits at their best.

Yet, it is worth trying in the long run as it is one of the QA tricks that can definitely enhance the quality of your product while saving time, effort and costs. Another thing you should bear in mind is that not every test that can be automated, should be. Mostly, it is the right balance between automated and manual testing that ensures sustainability and success.

So, it is only up to an experienced QA team to figure out what risk-based approach and tools to use in every single case to make your investment profitable. Thus, if you have further questions concerning your project, please contact us.