Why you need proactive testing strategies (and tips for getting started)

Quality assurance aims to prevent bugs — but many teams make the mistake of only using reactive testing strategies that only address bugs after they’ve been discovered.

When you rely solely on reactive testing strategies, you slow down development by addressing issues post development that you could have discovered earlier in the process. Incorporating proactive testing strategies helps streamline software development, reduce rework, and improve collaboration.

This doesn't negate the need for postmortem testing. Identifying post-development defects is crucial both for understanding when and why our system starts producing faulty code, and for not allowing defects on production.

The critical question is when testing should occur.

Postmortems vs premortems

Postmortems: after the incident

Many people are well-acquainted with postmortems. A postmortem is a reactive measure taken after a certain breakdown, where we scrutinize the possible factors that caused the failure and determine what alterations are needed to avoid such issues in the future. Postmortems include witnessing the failure during the support phase (after development), urgently addressing the problem, investigating potential causes, crafting alterations to lessen the chances of it happening again, and putting these changes into action.

Postmortems pros:

• It’s obvious what to analyze — the incident has already occurred
• It’s a very straightforward process

Postmortem cons:

• Open systems are not suitable for analysis: 5 whys don’t work well enough
• It’s too late — the incident already occurred, clients were affected, and the damage is done
• It's highly likely that a significant amount of work was misspent in the development process that resulted in the incident, so all those portions may need to be overhauled
• Premortems: before the incident
• The premortem is a more beneficial method where potential risks of failure are identified while the code is still being developed, way before it reaches the production stage. By catching a potential issue early on, premortems can reduce the need for corrections and relieve developers from the pressure of having to hastily fix urgent problems.
• Setting up a premortem is pretty straightforward: you and your team kick off a brainstorming session with the assumption that the feature has failed in production. Then, you try to imagine all the potential problems that could have led to this failure.
• Premortem pros:
• Incidents occur only in the imagination, no real damage is done to the clients
• Better quality, no rework

Premortem cons:

• Difficult to implement, blameless culture is a prerequisite
• Premortem excersise won’t catch all potential incidents

How to perform a premortem analysis and identify potential incidents

Setting up a premortem is pretty straightforward: you and your team kick off a brainstorming session with the assumption that the feature has failed in production. Then, you try to envisage all the potential problems that could have led to this failure.

The primary purpose of this meeting is to prompt the team to consider all possible mishaps. As a manager, you demonstrate that you are open to acknowledging failures and that your focus is on investigating the issues. People generally find it easier to agree rather than voice the risks, so when a manager acknowledges the failure and does not place blame, but instead appreciates the viewpoints of the team, it's more likely that team members will share their thoughts.

The outcome of a brainstorming session should be a list of potential reasons why the incident might happen. Then the team takes time to analyze what reasons are actually likely to happen, drafts a plan to address each reason, and then executes the plan.

Premortem is better done for bigger features, where the time needed for the activity is rationalized by the risk of potential incident.

In one of the companies I worked for, the graphs showed a peculiar case of a chat worker regularly disconnecting, reconnecting, and losing client sessions, particularly often in the mornings and evenings of one city: London. The app itself was working fine, and testing didn't show any issues. The issue was later found to be due to the London tube having no internet connectivity in the tunnels but good connection at the stations.

During development, everyone assumed that if the app on the phone is logged in and connected, it’d stay connected for quite a significant time. A premortem for chat feature might have helped us consider intermittent connectivity too.

Proactive requirements testing

Requirements testing is often done after the development stage is completed, but you can also do it before.

Performing initial requirements testing, like during the feature design or requirements testing stage, offers advantages akin to premortems. It makes the revamp of a feature more cost-friendly, aids in coaching engineers to produce superior quality code, and eases any needed process modifications. Early testing not only trims down development expenses, but it also makes the introduction of preventive QA steps more budget-friendly, thus diminishing the probability of repeated issues down the line.

The process is straightforward: whenever a feature is brought to development, first conduct requirements testing. This involves listing specified requirements, checking them for contradictions, and preparing a checklist for further testing. This ensures that during each subsequent stage of development and testing, the feature being built matches the defined requirements.

Think proactively and streamline software development

Testing is more efficient when it’s integrated into the entire software engineering process, rather than being a separate activity conducted after development is completed. Similarly, early testing activities are easier to coordinate and execute when the QA/QC engineer is part of the development team, rather than in a separate department.

References:

• https://www.uio.no/studier/emner/matnat/ifi/INF9181/h11/undervisningsmateriale/reading-materials/Lecture-10/post-mortems.pdf
• https://www.atlassian.com/incident-management/handbook/postmortems#postmortem-issue-fields
• https://www.uio.no/studier/emner/matnat/ifi/INF9181/h11/undervisningsmateriale/reading-materials/Lecture-10/post-mortems.pdf
• https://www.researchgate.net/publication/250884638_A_Review_of_Small_and_Large_Post-Mortem_Analysis_Methods