Embrace availability heuristics and balance your bias to become a better tester
The availability heuristic is a cognitive shortcut used to assess the likelihood or risk of events based on the ease with which instances can be recalled. If numerous examples readily come to mind, it is presumed that such events are common.
Qase Developer Advocate, Vitaly, touched on this topic in an article questioning whether bugs are really as prevalent as they seem or if it’s QA bias. He reflected on the tendency he sees in himself and other QA folks: they see defects and bugs everywhere.
Rease, Director of Content at Qase, does not come from a QA background and wanted to explore the idea that maybe, cognitive bias isn’t always a bad thing.
Together in this article we (Vitaly and Rease) explore the idea that availability heuristics can be a blessing and a curse in quality assurance work: QA professionals always seem to be encountering issues, but this ongoing experience can help them excel at their work.
Availability heuristics are part of everyday life
The availability heuristic (also known as availability bias) stems from a series of papers by Amos Tversky and Daniel Kahneman that examined heuristics and biases. Tversky and Kahneman examined the human tendency to rely on the first examples that come to mind as part of the decision-making process.
Let’s say you have the option to get coffee from Jose’s Coffee or Matilda’s Coffee. The last three times you went to Jose’s Coffee, your order was incorrect. Chances are, you’ll decide to pick up your latte from Matilda’s. Is the service at Jose’s Coffee always poor? If you’re relying on the availability heuristic, you’ll likely think so.
Now let’s explore a real world example of the availability heuristic for a QA professional. Years ago, Vitaly assisted his mom when she was having problems with an online banking app caused by internationalization (i18n) issues. When she switched the app to her native language, the form labels expanded and the input for the account number was inaccessible so she couldn’t enter the transfer recipient. Vitaly had to submit a support request to the bank to fix the issue.
During that same time, Vitaly was working on a system that could be affected by i18n issues. With his mom’s experience fresh in his mind, he decided to perform various checks on the system to ensure there were no similar i18n issues. Although the testing sessions confirmed that there were no such problems, Vitaly still decided to write some automated tests to cover the main screen functionalities so that, if any similar i18n issues were to arise, they would be detected before manual testing.
Availability heuristics are often seen as a problematic way of thinking
Like any cognitive bias, availability heuristics can significantly influence decision-making and judgment, often resulting in inaccurate assumptions. The availability heuristic leads individuals to overstate the likelihood of events based on their ability to recall specific examples.
Often, this mental shortcut is combined with the representativeness heuristic. Humans tend to create a stereotype or “rule of thumb” based on the people and situations represented in their memories. Both availability and representativeness bias can distort understanding and create incorrect assumptions about the prevalence or significance of certain events, personality traits, characteristics, etc.
Going back to the coffee shop example, imagine you pop into Jose’s Coffee — the shop that has made your order incorrectly the last three times you visited. A barista you’ve never seen before is preparing orders. Meanwhile, a barista you recognize as the person who messed up your latte last time is taking orders. Now, you have both the recent memories of incorrect orders (availability heuristic) and the link to someone who has previously prepared your order incorrectly, which introduces the representativeness heuristic.
In your mind, the recent experiences and similarities between both baristas represents confirmation that your coffee is going to be prepared incorrectly. In this case, this is a harmful (and potentially inaccurate) assumption as you have no logical reason to believe that this new barista will perform in the same way as the other.
Now let’s consider how this plays out in the QA world. In Vitaly’s situation, he chose to run i18n tests even though they were not originally planned. You might not think being extra cautious could be harmful, but consider this — what if Vitaly was so affected by the availability heuristic that he prioritized testing for i18n issues over testing an urgent release of a core feature. In that situation, his bias is problematic because if the core functionality doesn’t work, there is no point in testing for i18n issues.
Situations like these demonstrate the potential harm of relying on availability heuristics — leading to inaccurate stereotypes, unfair perceptions of situations or people, or illogical decisions. However, availability heuristics don’t always have to be problematic.
Availability heuristics can be used to your advantage in QA work
If we rely too heavily on availability heuristics, we’re in danger of making poor decisions born out of confirmation bias and inaccurate assumptions. However, the reality is — humans have thoughts and feelings that will inevitably influence our perceptions.
Embracing our human propensity for availability heuristics can help us mitigate its negative impacts and enhance our testing by leveraging its positive aspects — provided we know how to use availability bias to our advantage.
Imagine you’ve been working with a product for awhile and you’ve noticed that releases related to in-app notifications tend to break integrations with various project management tools. You don’t have any data to suggest that the upcoming release will cause issues, but the availability heuristic makes you feel like you have (anecdotal) evidence. You decide to preemptively set up additional testing to ensure that integrations continue to function properly and end up catching a few bugs in the process. In this situation, the availability heuristic helped you make a probability judgment and catch bugs before they hit production.
Availability heuristics can bring a diversity of experience and thought to the table. QA professionals often spot issues in various systems, from bus ticket purchasing to online shopping and banking. By observing these issues and considering potential causes, we may bring these cases to work and test for them.
When Vitaly encountered i18n issues in his mother’s banking app, leading him to do additional i18n testing at work, he was relying on the availability heuristic. However, he was also approaching testing with the end-user in mind. By considering how a user — his mother, in this case — would interact with an application that should function in multiple languages, he was able to take a proactive testing approach and prevent user discontent in the future.
Great testers rely on their instincts
Risk assessment in quality assurance can be challenging as we can only predict certain risks and their effects. Availability heuristics bring the most frequently recalled issues to the forefront and can provide a solid starting point for where, when, and what to test.
The key is to learn how to incorporate our knowledge of human behavior into our work and use availability heuristics to expand our predictions. For instance, if a recent release had significant problems with a certain feature, more attention and resources might be allocated to testing that feature in the current release. If the feature is still problematic, the heuristic has proven useful, ensuring that this functionality is adequately addressed.
Testers need to have well-developed instincts and sharing stories and narratives helps build those instincts. When software testers and quality assurance professionals share examples of critical bugs or issues, some form of bias is inevitable. But if we learn to use those experiences to diversify our thought processes, reinforce continuous learning, and optimize our approach to quality, we can use bias to our advantage.