BackstopJS vs Chromatic

Thinking of visual regression testing using Chromatic or an open source library like BackstopJS?

It can be time-consuming to scan each solution, so we collected some handy info that’ll make it easier to determine which tool is a good fit for your needs.

Let's compare Chromatic and BackstopJS by walking through features, workflows, and collaboration.

Are you building apps out of components?

There are plenty of visual regression testing tools out there but they don't work the same. Most are designed to test websites at the page level. They’re made for marketing pages, blogs, and news sites. BackstopJS falls in this category. It is designed to test pages (and is pretty good at it!).

The challenge that app teams discover is that visual testing a multi-state web app or SPA is very different than testing a collection of static pages. Web apps generate countless UI permutations based on user input and context. They're assembled from UI components.

Unfortunately, testing modern web apps built from components using old-school page testing techniques results in a flood of false positives that you have to dredge through to find real changes. Not to mention, the features for testing static sites aren’t that useful for componentized apps.

Chromatic was created because we were tired of complicated review flows (or none at all), false positives, a bunch of features we didn’t need, painful integrations, and test specs that bloated project repos.

Chromatic integrates seamlessly with Storybook to automatically track components from commit-to-commit, capture image snapshots in the cloud, and gives you a merge manager to help you review changes with your team.

Chromatic is built for teams

BackstopJS runs tests locally. While this is suitable for solo developers it's taxing for collaboration because each team member has to pull the code/screenshots just to run the tests. What's more, test running is time-consuming for large component libraries.

It takes a team to build and support a full-featured app. Like you, we understand that complexity grows and communication muddies the larger the team and project. That’s why Chromatic was built for collaboration from the start.

Chromatic captures and stores component screenshots at lightning speed in the cloud. Our online merge workflow helps you pull in stakeholders to review and confirm changes without needing to pull code, migrate, or maintain a dev environment.

Chromatic is accurate and lightweight

Most people testing on their local machine using Backstop encounter two pain points:

• The way UI is rendered varies between machines and operating systems even on the same browser. This results in a deluge of false positives. People mitigate this by lowering sensitivity thresholds of the diffing algorithm at the risk missing changes altogether.

• Screenshots stored alongside the code cause repo bloat. You can try storing them in the cloud but it's often not worth extra developer effort on configuration, hosting, and maintenance to keep screenshots in sync.

With Chromatic you never encounter these pitfalls. Every time you push code, it spins up identical machines that screenshot your components the same way every time. These screenshots are stored outside your repo in the cloud. They're indexed every commit across branches so you get an accurate history of how your component appears at any given time.

Chromatic is perfect for Storybook

Storybook helps frontend teams ship components. Chromatic automates visual testing those components.

As maintainers and sponsors of Storybook, we understand how integral Storybook is to app teams. That’s why we’re dead set on building a tool to improve your Storybook workflow without adding extra work.

Every story is a test

Chromatic tests map 1-to-1 with your stories. Whenever you update a story your tests are synced. This means visual tests are painless to keep up to date.

Your Storybook is uploaded automatically

Every time you push code, the components in your Storybook are uploaded to Chromatic. This means everyone on your team can participate in reviewing. No pulling code required.

No hassle branching and merging

Work on features together or separately. With branching and merging, you can make as many feature-branches as you need and seamlessly merge them back into production. It all happens behind the scenes, and just works. No configuration needed.

Chromatic makes reproductions easy

Chromatic doesn’t just capture an image of each component, it’s a living re-creation of component’s DOM, styling, and assets. Whenever you’re reviewing changes, you see high-fidelity components that are fully inspectable. This makes it painless to reproduce and confirm component changes.

Chromatic has common-sense pricing

BackstopJS is free and that's pretty appealing. But that price tag does not factor in the time sunk on configuration, maintenance, and stretching the project to fit the needs of modern web apps.

We reckon the hassle-free workflow boost you get immediately from Chromatic is a fantastic value.

Chromatic helps you visually test and maintain your component library in the cloud. Each time you push code, we instantly find visual changes, notify you, and help you review them with your team.

If you save just 3 dev hours each month then it pays for itself!

Ready to try Chromatic?

We hope this guide makes choosing the right visual regression testing tool a bit simpler.

Trying Chromatic is easy, install the addon and setup your project in less than a minute. Free plan. No credit card required. Sign up now.