The best way to check that your app works as it should, from start to finish, is end-to-end (E2E) testing. But there’s a catch with how they’ve traditionally been relied on. They skip the crucial step of verifying how apps actually look, which lets broken UI ship to prod unnoticed.
We've been working on a solution to this problem by creating a way of bringing Chromatic into your E2E tests. This lets you automate visual testing on every step of the E2E journey in Playwright and Cypress. Plus, it compiles the visual test cases into an interactive archive, so that you can debug the real page with a browser.
This post walks you through the entire E2E Visual Tests workflow in an example app called 'Pets in Pajamas'. Yup, it’s a social network for pets… in their pajamas.
Introducing Pets in Pajamas
Pets in Pajamas is the world’s most popular social network for making your adorable pets even more so by putting them in pajamas. People around the world rely on Pets in Pajamas’ patented AI pajamatization services every single day.
Unfortunately, Pets in Pajamas recently had a major setback. A bug in the pajama algorithm was leaving pets pajama-less or putting them in outfits that weren't pajamas.
Functional testing isn't visual
Now, we know what you’re thinking. ‘Pets in Pajamas: you had one job. How did y’all get it so wrong?'
You'd be right to suggest that Pets in Pajamas should have been running tests. In fact, they were doing that. They ran tests in isolation using Storybook and they ran E2E tests to assert that their services worked together. Their Playwright test for the pajama algorithm looked like this:
What they lacked was a solution for asserting their application's final visual output: the images themselves. This is what their users care about. The fact that all their tests pass is meaningless if there's a pet without pajamas at the end of it.
The solution: E2E Visual Tests
E2E Visual Tests is designed to highlight any unexpected visual changes in your application's UI. In the case of Pets in Pajamas, it will tell you when pets are – or aren't – wearing pajamas.
First, Chromatic will set visual baselines for what the UI should look like by capturing an image snapshot of the UI. Then, on future code changes, Chromatic will highlight when the UI differs from the baseline snapshots.
In this workflow, Chromatic highlights the visual differences so that devs can instantly spot the buggy image generation algorithm — even if it passed functional tests. That ensures the devs can make changes so that the app works and looks correct.
Try out E2E Visual Tests
E2E Visual Tests are available to try now in early access. All you need to do is swap Playwright’s
expect utilities for Chromatic’s. We’re also working on a Cypress implementation.
Sign up for early access by completing the form below or click here ≫
Recent updates in E2E Visual Tests
We added full support for Chromatic’s configuration options in Playwright and shipped bug fixes to reference and display assets.
- Set delays: Define a minimum amount of time to elapse before a snapshot is taken. This gives animations and other client-side set-up time to complete.
- Animation pausing: Chromatic will pause CSS animations in Playwright tests by automatically to prevent false positives. Customize this behavior by setting
- Adjust visual sensitivity threshold: Chromatic uses a threshold to determine whether to flag a visual change. By default,
diffThresholdis set to
.063, which balances high visual accuracy with low false positives. You can tweak sensitivity by passing your desired value.
- Customize anti-aliasing behavior: Chromatic detects anti-aliased pixels and auto-ignores them to prevent false positives. To flag every single pixel change, customize your settings with
- New capabilities for ignoring regions in a snapshot
- Our Chromatic Cypress Integration is in the works!