User acceptance testing (UAT): How to navigate the last development hurdle before production
If you asked a room of developers which step in the software development and testing process was the most difficult, you’d get many different answers. Some like the initial, exciting design process but loathe maintenance; some don’t care to design but revel in implementation. But if you asked that same room which step is the most stressful, the top answer will likely be user acceptance testing (UAT).
Unlike almost every other step, UAT is a bright line with a clear before and after. This is when someone determines whether all the effort you’ve put in so far has led to an app or feature worthy of production release. Even more nerve-wracking is that, in many cases, the people doing the testing don’t work at your company and might not even know how to code.
But the stress isn’t just localized to developers and coders. According to an Original Software study, 75% of companies run multiple UAT cycles, and 57% of those companies cite poor software quality as the reason they have to rerun those cycles.
UAT, done poorly, can be a stressful drain on resources, causing developers to rebuild and testers to rewrite test cases and rerun tests. UAT, done well, can help you perfect your release and feel confident pushing to production. It pays to learn how to do UAT well.
What is user acceptance testing (UAT)?
User acceptance testing (UAT) is a software testing process that occurs late in the software development lifecycle. Generally, UAT occurs after system integration testing (SIT) and regression testing.
Unlike many other software testing processes, which tend to be functional and require work from technical team members close to the code (usually developers, testers, or QA), UAT is often done by end-users or product owners. This is part of why UAT can be stressful: Developers and testers spend a lot of time working on a product and writing test cases, but “outsiders” (to varying degrees) influence what happens next.
The purpose of UAT
The purpose of UAT is to get the user’s perspective (or a proxy for their perspective) on the complete version of an app or feature.
The emphasis on the user perspective is important. Developers and testers work so close to the code for long periods of time that they may lose sight of things that can only be seen from a higher perspective.
If you’ve ever taken the famous inattentional blindness test, which asks viewers to count the number of times a small group of people passes a basketball back and forth, you’ll understand why a shift in perspective is necessary. When you focus on the basketball, you miss the gorilla. Similarly, when you focus on the code changes at hand, you can miss how features come together and what the overall user experience actually feels like.
However, not every company asks direct end-users to perform the user acceptance testing process. Some companies ask business analysts to perform UAT, and others ask stakeholders who work at the company but might not have worked directly on the product. Generally, the split is between companies that ask end-users and companies that ask the product team (writ large).
For example, an engineering director at a 4,000-person, publicly traded company, quoted by Gergely Orosz, reports that the company gives developers most of the manual and automated testing work but gives UAT — at least for new features — to the product team. In the same survey, a team lead at a startup with a QA team reports that developers write unit and integration test cases, QA handles E2E tests, and the product team, again, handles UAT.
When companies ask end-users to perform UAT, they might offer to pay them or provide some sort of incentive. For instance, one type of UAT, beta tests, is well-known in the video game industry.
UAT vs. SIT, regression, and TDD
UAT is a testing process that comprises varying amounts of other test types (which we’ll explain in the next section), and it sits adjacent to other testing processes, including SIT and regression tests — all of which can make defining UAT a little confusing.
The key is focusing on functional testing vs. non-functional testing.
With SIT, developers and testers examine how the system works as a whole to see whether all of the separately developed components come together to satisfy requirements and expectations. SIT typically consists of all functional tests, whereas UAT consists of all or mostly non-functional tests.
With regression testing, developers and testers check to see if any of the changes made since the last test introduced any new flaws. Again, regression testing is functional, whereas UAT is non-functional.
Additionally, with SIT and regression, testing is typically done by developers and testers close to the code, not teams and users outside of the people working on the code at hand.
There also tends to be confusion about UAT and test-driven development (TDD), not because they are similar but because some people new to TDD can assume it replaces UAT. To summarize a big topic, TDD is a software development methodology that inverts the typical process and asks developers to write unit test cases that fail, code to make the test pass, and then refactor — all happening iteratively over time.
At first glance, TDD can seem to replace the tests that typically happen after most feature development, but it doesn’t — especially when it comes to UAT.
As Erik Dietrich, founder of DaedTech, writes, “TDD is not a replacement for user acceptance testing. Someone who practices it does not believe that it’s a valid substitute for running the application and making sure that it does what the requirements state that it needs to do.”
Generally, despite the many approaches to development and testing and UAT, most teams believe you need to show the feature or app to a user or user proxy on some level.
5 types of UAT
There are five types of UAT, and different companies will employ different mixes of each and end-user different emphases in their test plans. When choosing, companies will weigh their priorities based both on which type of tests suit the product best and which users and user-proxies they have access to.
1. Alpha and beta testing
In these types of UAT, testers ask internal team members or outside users to use the product or feature and report any issues they have.
Internal team members typically perform alpha testing within development environments. Outside users, or at least user proxies, typically perform beta testing in the end-user’s environment (or a simulation of it).
In both cases, alpha and beta testers put the product through its paces to see if there’s anything wrong with how it came together. Like focus tests, the purpose is to give the product to a select group that responds in the same way a larger audience of users will.
2. Black box testing
In this type of UAT, testers check a system's functionality without knowing how it works internally. In the spirit of UAT, which is all about seeing how a user will react, the tester performing black box tests knows and cares as little about the product's inner workings as the typical user (i.e., not at all).
Testers focus on providing inputs to the product and checking the quality of the outputs. Testers typically simulate what users will do with the product, often emphasizing both expected and unexpected actions.
3. Contract acceptance testing
In this type of UAT, testers test the product against a list of predetermined criteria that teams source from the original requirements doc or specifications made before development.
As the name implies, this type of UAT is usually the best fit for development teams working on a contract basis via an agency or other partnership. When the business contract is first made, the company employing the developers will define precise requirements criteria so as to avoid scope creep and ensure, as well as possible, that the finished product will meet the agreed-upon criteria.
4. Regulation acceptance testing
In this type of UAT, testers check whether the product or feature meets the relevant regulatory and compliance requirements. Particular industries, such as healthcare with HIPAA, and particular regions, such as Europe with GDPR, will require either higher standards or stricter consequences for not meeting those standards.
5. Operational acceptance testing
In this type of UAT, testers check how effectively the product or feature will run in real-world conditions. More specifically, testers will check whether a range of operational features work well, such as how well the product handles:
- The installation process
- Load balancing and performance stress
- Backup and restoration
- Failover and recovery
Here, the criticality of the feature and the amount of expected traffic play big roles. Mission-critical features that will have tons of traffic and usage will require very careful operational acceptance testing.
Challenges of UAT
As we covered in the introduction, UAT done well can be a boon to the development and business teams alike: Developers can be confident releasing to production, and the business-side can feel comfortable knowing users will, for the most part, be satisfied.
Unfortunately, many teams don’t do UAT well, and the process can become something of a black hole, causing projects to fall out of scope and deadlines to slip. These mistakes often happen because UAT isn’t obviously complex, but it can quickly become difficult if you don’t stay ahead of some common challenges.
Scope creep
The biggest challenge with UAT complements its biggest advantage: When you’re performing UAT, you’re trying to see the product from a higher level, from a perspective you don’t normally have. Sometimes, this can result in the need for major changes — for both good and bad reasons.
If the initial requirements document was written hastily, many unseen requirements can emerge during UAT. Users can, quite fairly, ask for things the development team didn’t have planned. But if the requirements document didn’t cover everything, the development team might have to take that test data and build new components.
Sometimes, however, the feedback can be less urgent and more distracting, especially when UAT includes business-side stakeholders. This can take the form of a stakeholder insisting on a feature they didn’t originally request, one that could be added later, but one that they think is necessary for release. Here, UAT can become more political than technical, and developers might need to draw clear boundaries about scope creep and make clear what can be done now vs. what might need to wait for tomorrow.
The product isn’t ready yet
Another common challenge occurs when teams focus too much on chasing a product along the development lifecycle and not enough on ensuring each step is truly completed. In these cases, development teams can end up pushing a product to the UAT stage before it’s ready, resulting in a waste of time and resources.
Before a product is really ready for UAT, development teams need to ensure the testing team has the business requirements; the system, integration, and unit tests are complete; and the regression tests are done. Otherwise, UAT can’t begin or could result in discoveries that would have been better caught earlier in the process.
Ambiguous acceptance criteria
UAT tests are, for the most part, non-functional, but that doesn’t mean they should be vague or abstract. You might, for example, be testing the usability of a product, but the acceptance criteria have to go beyond “Is the product user-friendly?”
Effective acceptance criteria are specific, concrete, and clear. They direct testers to examine things like load time across various devices and settings. This is especially important when testing complex software that works across already complex systems.
Don’t just accept UAT — embrace it
Developers and testers often look at UAT with a little trepidation because it’s consequential. In the worst cases, it can mean rebuilding new components or building entirely new ones. It can be emotional as well — having a user point out a flaw that “should” have been obvious can be embarrassing.
But the very reason UAT can be stressful is why development and testing teams should embrace it and work toward doing it better. UAT is an important safeguard, and learning how to do it well benefits both the product and the people running the tests.