To automate or not to automate?

Jenna Charlton -

The soliloquy of test automation 

To automate or not to automate: that is the question;

Whether tis nobler in the release to risk

The bugs and anomalies of outrageous fortune,

Or arm ourselves with automated testing against a sea of unknowns…

Okay, okay, my version lacks the profundity of a 16-year-old Danish prince questioning the nature of morality and mortality, but the question is equally valid for us quality engineers. It feels like the answer to the age-old question of whether or not to automate shifts with the trends. Ultimately, the real answer from my perspective is, (as always) it depends.

It depends

Like it or not, the answer to whether or not to automate is always: it depends. What it depends on is unique to each organization and the context of the software under test. Contrary to what some will tell you, it’s impossible to automate everything. And to take this a step further, there’s no value in automating everything. Test automation is one of the many ways to guard organizations against the risk of bugs and unexpected behavior in their software. It’s impossible, however, to mitigate every risk, especially if there are risks that you can’t foresee. 

Or put another way — automation helps you test against the known unknowns, but automation will not uncover the risks associated with the unknown unknowns. 

Before I continue, I would be remiss if I did not transparently share that I believe strongly in automation. Meaningful and comprehensive automation strategies should be paired with thoughtful and technical manual testing. And when that testing is executed by skilled professionals who use proven techniques — that is how you get high quality, delightful software.

Since decisions about whether and what to automate are context dependent, how do you decide? There are three questions you should ask yourself when making decisions about when, what, and how to automate:

  1. What are your team’s capabilities?
  2. What is the context of your software?
  3. Where can automation provide your team the most value?

These decisions are a bit like the hero's journey, everyone’s path is different and everyone is starting at different points in the story. 

What are your team’s capabilities?

Setting up test automation requires a specific set of skills and these are not skills that are easily acquired. Ultimately, test automation is a software development project and needs to be developed with the same care and commitment to quality as the software it tests. It goes without saying that you will need SDETs or QAs with software development experience to set up a framework and begin a test automation project. This of course raises the question: what do you do if you don’t have a team with the right skills?

You have choices — the right one is going to depend on your team. Some options you may want to consider are:

  • Hire SDETs or automation engineers
  • Upskill your existing team
  • Outsource automation
  • Onboard a no-code or low-code automation tool
  • Work with your development team to have them start building your automation

Only you and your organization can determine which one is right for you, and the decision making will include a number of factors. 

Your team: What’s the aptitude of your existing team? Are they likely to be able to learn the needed skills for automation? How about their capacity? Do they have time to learn the needed skills or will this impact your ability to release new features?

Your budget: What’s the budget for automation? Does the cost need to be predictable and fixed or is there an option for higher upfront costs? 

The organization: How much organizational support for automation do you have? Will the developers step in to help? 

Once you’ve considered your team, your budget, and your organizational support, you’ll be ready to identify the best route to take to automation. 

What is the context of your software?

The context of your software has an impact on all of your testing decisions. For example, if your software runs heart monitors, nearly exhaustive testing is likely required. In ecommerce, on the other hand, defects are undesired, but no one will die if they encounter a bug in production. The question of context is the same for the devices you need to cover. Do you need to cover web and mobile, multiple browsers, and different device types? All of these questions should be decision points in what you automate. 

Where can automation provide your team the most value?

I’ve already established that automation should be viewed as a partner and a resource to your team and that it’s impossible to automate everything, So if you can’t automate everything, what should you automate? 

My belief is to first automate the tests you need to run most often, the tests that are most critical, and the tests that are the hardest to execute manually. In other words, automate regression and automate the hard stuff. I know, easier said than done right? The good news is that once you know how to make these decisions and align them to your overall test strategy, it becomes much easier to get started and stay the path towards growing your automation suites. 

Risk-based: If you use a risk-based test strategy, you should already have a matrix that lists your tests from highest to lowest risk. Your initial focus should be on automating as many of the highest-risk tests as possible. This of course takes into account that not everything can be automated. The good news is that automating the highest-risk tests gives you more time for testing what you can’t automate. 

Safety-critical: For most organizations that build safety-critical software, there is some sort of official guidance from a regulatory body or oversight board inside or outside of the organization. Your first resource in establishing what to automate should be the documentation from the regulators. Once you’ve established what you have to do based on their guidance, you can establish what will best serve your team. For example, you might automate complicated or time-consuming tests that you run on a regular basis to help free up more time for your team. 

Everyone else: Your testers are your best resource in making these decisions. Your testers likely know which regression suites they run most often and which are the hardest to execute. Help your testers develop their skills in prioritization by having them identify which tests being automated would be the biggest time and effort savers for them.

Other valuable resources when determining what to automate are your systems and metrics. For instance your test management system (TMS) can provide data on tests that are run most often, fail most often, or are used in multiple suites. Additionally, your TMS and your requirements management system can help you maintain traceability, which in turn will help you see which tests are the most important to automate first.

Is ‘to automate or not to automate’ truly the question?

Ultimately, no — the question is not whether or not to automate. The real questions are what method to use and how you can leverage automation to support your overall quality strategy.