Press "Enter" to skip to content

How to run visual regression tests using BackstopJS in 5 simple steps?

Vladimir Simonovski

Visual regression tests or visual testing is an activity performed by the testers to compare different versions of the screenshots taken from the application. A typical visual test would look like this:

visual regression tests using BackstopJS
Visual test flow
  • Take a screenshot of the application you are testing
  • Save the screenshot and set it as a baseline
  • Deploy the newest application version of the UI
  • Open the application where you took the screenshot and compare it with the older screenshot
  • If there are changes, save the newest screenshot as a baseline, if not, the initial baseline would be still valid

The flow looks pretty straightforward right? Well, yes if you are testing a couple of pages only once.

But what if we need to test a large quantity of application web pages or specific sections every single time there is a change on the UI? Doing that manually over and over again would take a lot of time for the testers to be able to have the needed resources to focus on other aspects of testing.

Automated visual regression tests using BackstopJS are one way to automate this process. The benefits are:

  • Automatically compare screenshots
  • Multiple viewports compatibility
  • Screenshots from all webpages or specific sections
  • Easy setup
  • Comprehensive UI report

What is BackstopJS?

BackstopJS is an automated test framework that is written in JavaScript. It servers the purpose of automating visual regression tests, it is simple to set up and configure, and easy to understand. It has many components including the ability to render the DOM and all the elements through the command line, the ability to interact with the page as well as compare images in pixel differentiation.

Let’s show you how you can automated visual regression tests using BackstopJS in 5 simple steps!

Install BackstopJS

Assume that you already have Chrome and node.js installed on your computer, lets see how you can install BackstopJS. Globally install it on your computer using this command:

npm install -g backstopjs

Setup and configure BackstopJS

Create a new folder structure, navigate to the project directory and run:

backstop init

This command creates a folder called backstop_data and also, will create a JSON file called backstop.json.

The JSON file will serve as the main confrontation for our tests. This would be the outcome of the command above:

BackstopJS folder structure

The initial configuration created in the backstop.json file is:

{
  "id": "backstop_default",
  "viewports": [
    {
      "label": "phone",
      "width": 320,
      "height": 480
    },
    {
      "label": "tablet",
      "width": 1024,
      "height": 768
    }
  ],
  "onBeforeScript": "puppet/onBefore.js",
  "onReadyScript": "puppet/onReady.js",
  "scenarios": [
    {
      "label": "BackstopJS Homepage",
      "cookiePath": "backstop_data/engine_scripts/cookies.json",
      "url": "https://garris.github.io/BackstopJS/",
      "referenceUrl": "",
      "readyEvent": "",
      "readySelector": "",
      "delay": 0,
      "hideSelectors": [],
      "removeSelectors": [],
      "hoverSelector": "",
      "clickSelector": "",
      "postInteractionWait": 0,
      "selectors": [],
      "selectorExpansion": true,
      "expect": 0,
      "misMatchThreshold" : 0.1,
      "requireSameDimensions": true
    }
  ],
  "paths": {
    "bitmaps_reference": "backstop_data/bitmaps_reference",
    "bitmaps_test": "backstop_data/bitmaps_test",
    "engine_scripts": "backstop_data/engine_scripts",
    "html_report": "backstop_data/html_report",
    "ci_report": "backstop_data/ci_report"
  },
  "report": ["browser"],
  "engine": "puppeteer",
  "engineOptions": {
    "args": ["--no-sandbox"]
  },
  "asyncCaptureLimit": 5,
  "asyncCompareLimit": 50,
  "debug": false,
  "debugWindow": false
}

The viewports are different screen resolutions that the visual test would run for a particular website URL. You can re-configure them. Add viewport for desktop screens:

   {
      "label":"Desktop",
      "width":1280,
      "height":1024
   }

The paths object is where our screenshots are saved. The folders are created after test execution.

bitmaps_reference is the folder where our baseline screenshots are saved. bitmaps_test is the folder where comparisons of two screenshots take place and it also stores any potential differences highlighting them with squared shape.

The last thing we need to do here is adding the scenarios. As you can see the initial scenario is the default one and it is created after we initialize the project. We can change the label and URL to what we want. We can add a new scenario for a specific URL within the particular website or a totally different website URL. For this post, we will leave it as it is.

Now we’re ready to run the test!

Run the initial test

The command for running the test is:

backstop reference

The URL that we are hitting is https://garris.github.io/BackstopJS/ and we have three viewports, phone, tablet, and desktop resolutions. This command will save three screenshots in bitmaps_reference folder for the three different viewports. The output of the command is:

COMMAND | Executing core for "reference"                             
  clean | backstop_data/bitmaps_reference was cleaned.               
createBitmaps | Selected 1 of 1 scenarios.                           
CREATING NEW REFERENCE FILE                                          
Cookie state restored with: [                                        
  {                                                                  
    "path": "/",                                                     
    "name": "yourCookieName",                                        
    "value": "yourCookieValue",                                      
    "expirationDate": 1798790400,                                    
    "hostOnly": false,                                               
    "httpOnly": false,                                               
    "secure": false,                                                 
    "session": false,                                                
    "sameSite": "no_restriction",                                    
    "url": "https://.www.yourdomain.com"                             
  }                                                                  
]                                                                    
CREATING NEW REFERENCE FILE                                          
Cookie state restored with: [                                        
  {                                                                  
    "path": "/",                                                     
    "name": "yourCookieName",                                        
    "value": "yourCookieValue",                                      
    "expirationDate": 1798790400,                                    
    "hostOnly": false,                                               
    "httpOnly": false,                                               
    "secure": false,                                                 
    "session": false,                                                
    "sameSite": "no_restriction",                                    
    "url": "https://.www.yourdomain.com"                             
  }                                                                  
]                                                                    
CREATING NEW REFERENCE FILE                                          
Cookie state restored with: [                                        
  {                                                                  
    "path": "/",                                                     
    "name": "yourCookieName",                                        
    "value": "yourCookieValue",                                      
    "expirationDate": 1798790400,                                    
    "hostOnly": false,                                               
    "httpOnly": false,                                               
    "secure": false,                                                 
    "session": false,                                                
    "sameSite": "no_restriction",                                    
    "url": "https://.www.yourdomain.com"                             
  }                                                                  
]                                                                    
Browser Console Log 0: JSHandle:BackstopTools have been installed.   
SCENARIO > BackstopJS Homepage                                       
Browser Console Log 0: JSHandle:BackstopTools have been installed.   
SCENARIO > BackstopJS Homepage                                       
x Close Browser                                                      
Browser Console Log 0: JSHandle:BackstopTools have been installed.   
SCENARIO > BackstopJS Homepage                                       
x Close Browser                                                      
x Close Browser                                                      
                                                                     
Run `$ backstop test` to generate diff report.                       
                                                                     
      COMMAND | Command "reference" successfully executed in [5.532s]

And three different screenshots are saved in the bitmaps_reference folder that just got created during execution (phone, tablet, and desktop).

backstopjs
Baseline screenshots

Compare two screenshots

Since we already saved our baseline images for different device resolutions, we are ready to compare those baseline images with the newest ones. The command for image comparison would be:

backstop test

The outcome of this command is:

      COMMAND | Executing core for "report"
      compare | OK: BackstopJS Homepage backstop_default_BackstopJS_Homepage_0_document_0_phone.png
      compare | OK: BackstopJS Homepage backstop_default_BackstopJS_Homepage_0_document_1_tablet.png
      compare | OK: BackstopJS Homepage backstop_default_BackstopJS_Homepage_0_document_2_Desktop.png
       report | Test completed...
       report | 3 Passed
       report | 0 Failed
       report | Writing browser report
       report | Resources copied
       report | Copied json report to: ~\backstop_data\bitmaps_test\20210201-123652\report.json
       report | Copied jsonp report to: ~\backstop_data\html_report\config.js
      COMMAND | Executing core for "openReport"
   openReport | Attempting to ping
   openReport | Remote not found. Opening backstop_data\html_report\index.html
      COMMAND | Command "report" successfully executed in [4.538s]
      COMMAND | Command "test" successfully executed in [10.109s]

This is showing us that the UI has not been changed and the images are the same. That’s why we have 3 tests passed for all of the screenshots.

Three images are created under bitmaps_test folder.

Comparison screenshots

Since those are the same as the previous ones then we conclude that our UI application page has not been affected by any change made by the developers.

Repot is automatically generated:

backstopjs test report
BackstopJS test report

The report shows all the tests that have been executed as well as all of the screenshots that’ve been compared.

Save new screenshot as a baseline

Since there are no changes in the UI, we don’t need to save the newest images but just for the purpose of showing how you can store those screenshots we are going to save them, using the command:

backstop approve

The output from the command above would look like:

save images
Save new screenshots output

All of the newest screenshots from the bitmaps_test will be promoted to bitmaps_reference folder and would serve us as a baseline for upcoming test comparisons. This would be the case if the design has changed. In some situations, the UI may be broken and after we notice that the images are different, we need to coordinate with the developers and address the issue accordingly. In that case, we would not save the new screenshots since it is a bug, not a change request or a feature.

Conclusion

Visual regression tests have never been so easy and straightforward. The constant effort provided by experts in the field and from the community itself, allows us to take advantage of the tool’s functionalities and benefits. Visual regression tests using BackstopJS is one way of automating these kinds of tests. There are other tools for this purpose like Aplitools or Wraith. What tools would you use is based on your predefined strategy within the project.

Share This Post

Latest Posts

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.