A guide to sanity testing

Sanity testing is a specific type of software testing that serves as a litmus test for small changes or bug fixes within a software system. Its primary goal is to verify that these modifications:

• Work as intended
• Do not introduce unexpected side effects
• Do not disrupt existing functionality

Sanity testing, a subset of regression testing, typically occurs after smoke testing. It is specifically conducted on software builds that demand immediate deployment to production — especially when addressing critical bugs. This testing acts as a crucial checkpoint to decide whether to proceed with further testing. If the sanity test fails, the testing team rejects the build, saving valuable time and resources.

Purpose of sanity testing

The primary objective of sanity testing is to verify that the tested functionality within the program operates as intended and that recent updates have not introduced any defects. In other words, sanity testing ensures that minor changes or bug fixes do not disrupt the existing functionality.

How sanity testing differs from smoke testing and regression testing

Sometimes sanity, smoke, and regression testing are used somewhat interchangeably. While they share similarities, sanity testing, smoke testing, and regression testing serve distinct purposes in the testing process.

Sanity testing vs smoke testing

Sanity testing checks if the program works after changes have been made to the code. Smoke testing helps ensure that the most important software functions work correctly. Both of these types of testing are used to check the basic functionality of the application, but there are three key differences:

• Scope: Sanity testing usually has a narrower scope than smoke testing. Sanity testing checks specific functionality; for example, whether a certain function works after a bug fix. Smoke testing, on the other hand, has broader and more thorough coverage, checking the most important application functions.
• Depth: Sanity testing is a superficial check of one or more specific parts of the application. It is not intended to identify deep problems or errors in the system. Smoke testing, on the other hand, is a more thorough test that checks critical functionality.
• Purpose: The main purpose of sanity testing is to confirm that a small change in the system has not caused any unforeseen side effects and has not disrupted existing functionality. Smoke testing, on the other hand, is used to check the overall functionality of the system and ensure its stability and readiness for further testing.

The main difference between sanity and smoke testing is that sanity testing checks one or more specific parts of the application, while smoke testing checks critical functionality regardless of the new feature, change, or fix.

Sanity testing vs regression testing

Sanity testing and regression testing are two essential components of software testing, each serving distinct purposes in ensuring the quality and reliability of software products. While both types of testing share similarities, they also exhibit significant differences in terms of their scope, purpose, execution, and depth.

Sanity testing process

The primary purpose of conducting sanity testing is to identify the presence or absence of potential defects in newly added code or incorrect outcomes resulting from its operation, including any impact on the functionality of existing components. Typically, the sanity testing process consists of three main stages:

• Identification: During this stage, newly added functions and components, as well as changes in existing code resulting from bug fixes, are identified.
• Evaluation: The newly implemented components are scrutinized and adjusted to verify if they operate as intended according to the specified requirements.
• Testing: A small portion of the new components undergo comprehensive testing. The test fails if the newly implemented components fail to meet the requirements. Conversely, if they fulfill the requirements, the sanity test is deemed successful.

How to perform sanity tests

Unlike other software testing methods, sanity testing does not rely on an extensive array of techniques or scenario writing. Instead, it follows key steps: identification, evaluation, and testing. During these steps, it's crucial to determine the changes made in the product, verify them, and select associated functions. If these functions operate in line with requirements, the release can proceed to further testing. To enhance the efficiency of sanity tests, consider the following practices:

• Clearly define scope by establishing which specific areas of the system will undergo testing and identify any specific requirements or constraints that need consideration.
• Utilize automation tools: they can simplify the sanity check process, reducing the time and effort needed to complete it. Automation speeds up testing and reduces reliance on regression testing.
• Carefully analyze test results to identify areas that may require additional testing. This analysis helps ensure thorough coverage and identifies potential issues in the early stages of development.

Sanity testing can be conducted manually or automatically, each with its own advantages and disadvantages. Manual testing is suitable when resources and time for testing are limited, and changes are minor. However, it requires testers with high-level expertise in software functionality.

Automated testing involves using automated testing tools and frameworks to verify basic application functions. Automated tests are typically executed with each new build or release, ensuring consistency and efficiency.

In practice, sanity tests are often automated, especially since they are a subset of regression tests. Automating these tests helps save time and resources, maintaining the quality of the software application. To automate sanity testing effectively:

• Identify software functions which require testing
• Develop test cases covering specific functionality of the software application
• Select appropriate tools for automating the tests
• Run automated tests and analyze results. If a test fails, investigate the root cause, fix the issue, and report identified bugs to the development team, providing detailed information through screenshots and test logs.

  Sanity testing examples

Below are two examples of sanity testing, one manual and one automated.

Manual sanity testing

Scenario: verifying the login functionality of a web application after a minor code change or build deployment

Steps:

a. Open the web application in a web browser.

b. Locate the login page or section.

c. Enter a valid username and password combination.

d. Click the "Login" button or submit the form.

e. Verify that the user is successfully logged in by checking for the presence of the user's profile information or a welcome message.

f. Log out of the application

g. Repeat steps c-f with an invalid username and password combination to ensure that the application denies access to unauthorized users.

By following these steps, you can quickly confirm that the core login functionality is operating as expected after the code change or build deployment. If any issues arise during this process, they may indicate a potential regression or defect requiring further investigation.

Automated sanity testing

Scenario: verifying the search functionality of an ecommerce website after a minor code change or build deployment

Steps: 

a. Launch the automated testing framework and navigate to the ecommerce website's homepage 

b. Locate the search input field using the appropriate web element locator (e.g., CSS selector, XPath, etc.)

c. Enter a known product name or keyword into the search input field

d. Trigger the search action (e.g., click the search button or press Enter)

e. Verify that the search results page is displayed and contains at least one product matching the search criteria

f. Optionally, assert that the search results are sorted correctly or that specific filters are applied correctly. This automated sanity test can be written using the testing framework's scripting language (e.g., Java, Python, JavaScript) and executed as part of a continuous integration/continuous deployment (CI/CD) pipeline or scheduled to run at regular intervals.

Both manual and automated sanity testing are important in ensuring that the core functionalities of a software application are working as expected after code changes or build deployments. Manual testing allows for quick visual verification and can catch issues that may not be easily covered by automated tests, while automated testing provides a faster and more consistent way to verify specific functionalities and can be seamlessly integrated into the development workflow.

When to perform sanity testing

Sanity testing should be performed in the following cases:

• When there are minor changes in the code
• When a build is obtained after a series of regressions
• After fixing bugs
• Before deployment to production

Usually, after the release of a new application build, smoke testing is required, as smoke testing guarantees that you have stable builds. If the smoke testing is successful, then sanity testing is performed. If sanity testing also succeeds, then more in-depth testing is carried out, such as functional and/or regression testing.

This approach helps prevent further problems and ensures high software quality. Without sanity testing, if a full testing cycle is run, the testing efforts will most likely be in vain if a new release is required. The advantages of sanity testing outweigh the disadvantages. Sanity checking allows you to avoid unnecessary work by showing whether additional tests need to be performed. This saves time for the testing team and simplifies the process by avoiding formal bug reporting.

Advantages and disadvantages of sanity testing

Advantages of sanity testing

• Speed: Sanity testing does not necessitate documentation, enabling quick identification of defects or assurance of their absence. This agility is particularly beneficial when testing time is limited, providing prompt feedback to swiftly address critical issues.
• Efficiency: It serves as an effective means to verify that minor changes or modifications to the system have not induced unforeseen side effects or disruptions to existing functionality.
• Focus: Sanity testing enables testers to concentrate on specific areas of the system, ensuring that alterations made therein do not result in unexpected consequences.
• Cost-effectiveness: Generally, sanity testing requires minimal resources and time for completion. Detecting software issues during sanity testing leads to the rejection of defective builds, saving time and effort that would otherwise be spent on regression testing. Moreover, it aids in identifying significant issues early in the development cycle, thereby averting costly rework in later stages.
• Increased productivity: By allowing developers and testers to focus on more complex test scenarios, sanity testing enhances overall productivity and reduces the time needed for full regression testing.

Disadvantages of sanity testing

• Limited scope: Sanity testing focuses on a subset of functional capabilities of the application, potentially overlooking issues or errors outside its scope.
• Incomplete testing: It provides only a preliminary assessment of the software's performance and quality, failing to offer a comprehensive view of its functionality.
• Possibility of false positives: Due to its rapid nature, sanity testing may yield results indicating proper functionality when defects or issues exist undetected.
• Dependence on experience: The effectiveness of sanity testing relies heavily on the experience and expertise of the tester, necessitating a profound understanding of the software's critical functions for accurate testing.

The indispensable role of sanity testing in modern software development

Sanity testing stands as a pivotal practice within the software development lifecycle, enabling teams to swiftly and economically validate their product's integrity post-code modifications. Its primary focus lies in ensuring that the core, critical functionality of the application remains unaffected, prioritizing functionality over exhaustive bug detection.

The agility and efficiency inherent in sanity testing render it invaluable, particularly in time-sensitive scenarios necessitating immediate deployment, such as critical bug fixes. This adaptability proves especially crucial in today's fast-paced, iterative software development landscape.

Crucially, the cost-effectiveness of sanity testing further enhances its widespread adoption. By preemptively identifying issues and averting extensive rework, it allows organizations to optimize their testing efforts and resources, ultimately resulting in higher-quality software and improved return on investment.

In summary, the unique amalgamation of focus, speed, and cost-effectiveness positions sanity testing as an indispensable element of a comprehensive software testing strategy. As software systems grow in complexity, the ability to swiftly validate core functionalities becomes increasingly vital, solidifying sanity testing's role as a critical safeguard against regressions and a driving force behind software quality and delivery.