Sign inSign up

Putting Pets in Pajamas with E2E Visual Tests

Validate functionality and UI in one workflow

loading
Joe Vaughan
@joevaugh4n

The best way to check that your app works as it should, from start to finish, is end-to-end (E2E) testing. But there’s a catch with how they’ve traditionally been relied on. They skip the crucial step of verifying how apps actually look, which lets broken UI ship to prod unnoticed.

We've been working on a solution to this problem by creating a way of bringing Chromatic into your E2E tests. This lets you automate visual testing on every step of the E2E journey in Playwright and Cypress. Plus, it compiles the visual test cases into an interactive archive, so that you can debug the real page with a browser.

This post walks you through the entire E2E Visual Tests workflow in an example app called 'Pets in Pajamas'. Yup, it’s a social network for pets… in their pajamas.

Sign up for early access to E2E Visual Tests »

Introducing Pets in Pajamas

Pets in Pajamas: a gallery of pets in pajamas

Pets in Pajamas is the world’s most popular social network for making your adorable pets even more so by putting them in pajamas. People around the world rely on Pets in Pajamas’ patented AI pajamatization services every single day.

The problem

Unfortunately, Pets in Pajamas recently had a major setback. A bug in the pajama algorithm was leaving pets pajama-less or putting them in outfits that weren't pajamas.

Two photographs. One shows a pug in pajamas (labelled 'Yes pajamas'). The other shows a pug in a harness (labelled 'No pajamas').

Functional testing isn't visual

Now, we know what you’re thinking. ‘Pets in Pajamas: you had one job. How did y’all get it so wrong?'

You'd be right to suggest that Pets in Pajamas should have been running tests. In fact, they were doing that. They ran tests in isolation using Storybook and they ran E2E tests to assert that their services worked together. Their Playwright test for the pajama algorithm looked like this:

What they lacked was a solution for asserting their application's final visual output: the images themselves. This is what their users care about. The fact that all their tests pass is meaningless if there's a pet without pajamas at the end of it.

An animation showing the diff test for the Pets in Pajamas algorithm.

The solution: E2E Visual Tests

E2E Visual Tests is designed to highlight any unexpected visual changes in your application's UI. In the case of Pets in Pajamas, it will tell you when pets are – or aren't – wearing pajamas.

First, Chromatic will set visual baselines for what the UI should look like by capturing an image snapshot of the UI. Then, on future code changes, Chromatic will highlight when the UI differs from the baseline snapshots.

Pets in Pajamas' E2E Visual Tests inside Chromatic.

In this workflow, Chromatic highlights the visual differences so that devs can instantly spot the buggy image generation algorithm — even if it passed functional tests. That ensures the devs can make changes so that the app works and looks correct.

Try out E2E Visual Tests

E2E Visual Tests are available to try now in early access. All you need to do is swap Playwright’s test and expect utilities for Chromatic’s. We’re also working on a Cypress implementation.

Sign up for early access by completing the form below or click here ≫


Recent updates in E2E Visual Tests

We added full support for Chromatic’s configuration options in Playwright and shipped bug fixes to reference and display assets.

  • Set delays: Define a minimum amount of time to elapse before a snapshot is taken. This gives animations and other client-side set-up time to complete.
  • Animation pausing: Chromatic will pause CSS animations in Playwright tests by automatically to prevent false positives. Customize this behavior by setting pauseAnimationAtEnd.
  • Adjust visual sensitivity threshold: Chromatic uses a threshold to determine whether to flag a visual change. By default, diffThreshold is set to .063, which balances high visual accuracy with low false positives. You can tweak sensitivity by passing your desired value.
  • Customize anti-aliasing behavior: Chromatic detects anti-aliased pixels and auto-ignores them to prevent false positives. To flag every single pixel change, customize your settings with diffIncludeAntiAliasing.

What’s next?

  • New capabilities for ignoring regions in a snapshot
  • Our Chromatic Cypress Integration is in the works!

Did this article help you?

Get free UI development guides and tutorials like this emailed to you.

4,472 developers and counting

We’re hiring!

Join the team behind Storybook and Chromatic. Build tools that are used in production by 100s of thousands of developers. Remote-first.

View jobs

Popular posts

Introducing Chromatic’s integration for Cypress visual testing

Extend Chromatic’s visual tests into your Cypress workflow
loading
Joe Vaughan
Product
PricingAboutJobsTerms of ServicePrivacyStatusSecurity • SOC 2Contact Sales
Chromatic
© Chroma Software Inc. Made by the maintainers of Storybook.