Developing a custom cypress plugin for resilient visual tests

This week I was working on a plugin for Cypress and I want to share my experience with you. The why, what and how.

Why developing another Cypress plugin?

Visual Testing will become a crucial part of our testing strategy. We can (currently) not use a third party provider as some parts are rendered with webgl and the tests must run on premise. So we tried several existing plugins, but found out that they will not run reliable in our ci. I read about how Cypress improves stability for their DOM based commands and assertions with Retry-ability and (lately) Test Retries. Existing visual testing plugins are not based on this concepts and have problems implementing them (e.g. cypress-image-snapshot). So let's build our own.

What are we going to do?

This plugin is going to use cy.screenshot to take a screenshot and compare it to a baseline using pixelmatch. The only special thing is, that it will continue taking screenshots and comparing them until a timeout is reached. We are going to use the existing defaultCommandTimeout therefore.

One thing to take care of is that cy.screenshot() generates a unique filename for every attempt. This is where most existing plugins fail as they either just pass on the second attempt (new name = no baseline) or immediately fail.

How to develop your own Cypress plugin?

There are some Cypress basics you should know. Most code executed by Cypress runs directly in the browser, e.g. tests, command, everything in /support. But some tasks will run in node which are located under /plugins. We will need both:

• a command that fires cy.screenshot and our custom tasks
• the tasks itself that use pixelmatch to compare the images 

One side note: If you are developing a command that will be followed by an assertion, e.g. cy.get("something").should("contain", "some text") you will need to retry the following assertion. Read about cy.verifyUpcomingAssertions(). This is not what we are going to do here.

Here are some steps that this plugin will execute:

1. The command cy.shouldMatchBaselineImage() is called from a test step passing a name and (optional) screenshot and pixelmatch options.
2. The command starts a timeout using Cypress.config("defaultCommandTimeout") as value.
3. A function is called which will be looped as long as the timeout is still running and the comparison fails.
4. This function first calls cy.task("prepareForScreenshot") which will set a flag to recognize the following screenshot as relevant.
5. A Cypress native screenshot is taken which fires a on("after:screenshot",()=>{}) hook. We store the screenshot details in the plugin scope here If the "prepare flag" is set.
6. Then the updateBaselineImage task is called which uses the screenshot details, reads the screenshot image and its stored baseline, compares them and returns the pixel difference.
7. Based on the result and the timeout the command will either pass, retry, throw an error or fail. It is helpful to use some Cypress.log() and reduce our future debugging times.

That's it for now. I will probably publish the plugin when it has proven itself. Looks very promising for now.🤞Let me know if you have any questions!