BackstopJS vs Chromatic

Thinking of visual testing using Chromatic or an open source library like Backstop? It can be time-consuming to scan each feature of every tool, so we gathered some handy info that’ll make it easier to decide which visual testing tool is right for you.

Let's compare Chromatic and BackstopJS by walking through features, workflows, and collaboration.

Jump to section

• Can you set up Backstop by yourself?
• Is Backstop visual testing flaky?
• Is Backstop reliable?
• Is Backstop fast?
• How does Backstop handle branching and merging for baselines?
• How do you debug tests in Backstop?
• How do you collaborate in Backstop?
• What browsers does Backstop support?
• What does Backstop cost vs Chromatic?
• How does Chromatic visual testing integrate with my tech stack?
• Chromatic is perfect for Storybook
• Ready to try Chromatic?

💡 Why should you trust us? Our team has decades of experience in the open source community. We maintain Storybook, an industry-standard open source tool for frontend developers.

Can you set up Backstop by yourself?

Yes, you'll need experience with dev ops. Backstop provides the basic building blocks of a visual testing solution, but you need to stand up the infrastructure and scale it yourself.

Chromatic is designed for developers to get started in 90 seconds. Best-practice defaults come out of the box to prevent misconfiguration. In as little as three lines of code, you’ll be up and running. You don’t need to be an expert to start visual testing.

Is Backstop visual testing flaky?

It depends. Backstop visual tests are only as reliable as the code being tested. Your visual tests may be flaky if your UI renders inconsistently. For example, with randomized data, flashes of un-styled content, or slow loading resources.

With Backstop, since you own the visual testing infrastructure, you’re responsible for debugging rendering deep in the stack. For solo devs testing on their local machine, this might be a reasonable tradeoff. But as your team scales to production workloads, you may have to dedicate your own engineers to “own” the service.

Professional visual testing tools mitigate flake in a number of ways. With Chromatic, we prevent flake using a standardized browser environment that auto-detects UI loading, painting, and reflowing to determine the most reliable time to snapshot. We believe this prevents flake before it can affect your tests. Our team monitors visual test consistency so you don’t have to. Chromatic has run over 7.3 billion tests across thousands of companies.

Is Backstop reliable?

It depends. Backstop is an open source tool which means your team is responsible for infrastructure reliability end to end. Any time you update the infrastructure with new browsers, you may encounter false positives. As your usage fluctuates, you’ll also have to spin up CI processes and optimize the balance of RAM/CPU vs price vs reliability.

Chromatic has 99.9% uptime. It frequently releases infrastructure upgrades to increase performance, stability, and bump browser versions. So often that we invented “upgrade builds”, a feature that allows you to migrate UI test baselines between infrastructure versions without test flake or false positives.

Is Backstop fast?

It depends. The faster your tests run, the faster you iterate. When comparing visual testing tools, parallelization is the key differentiator for speed. Running tests in sequence can be up to 30 minutes slower than running tests in parallel.

When using Backstop, speed is relative to how much money you want to spend on infrastructure. You manage parallel test runs yourself by allocating additional CI build processes. For each concurrent process, you’d need about 105MB RAM.

Chromatic’s infrastructure is effectively able to run your entire test suite simultaneously and optimizes for fastest test times. Every plan we offer includes the fastest possible test speeds.

To give a benchmark for speed, Chromatic can render code, take a snapshot, and find visual differences for 2000 tests in less than 2 minutes. If you’re using Chromatic’s TurboSnap, you can further speed up your tests by 81% for the same coverage.

How does Backstop handle branching and merging for baselines?

A baseline represents what a test is supposed to look like. How a visual testing tool manages baselines is important to consider because you may have to change the way you work in order to use the tool.

Backstop doesn’t track baselines between branches or merges automatically. When you update a baseline, it saves the snapshot to a directory. You then check the directory with all your snapshots into your Git repo (which can be 100s of megabytes). This approach gets complicated when working with branches because merge conflicts between baselines makes it tough to know the real source of truth.

Chromatic tracks baselines using Git history. It mirrors how developers track code changes. Your baselines persist through branches and merges without you having to change the way you work. This allows entire engineering organizations to work on features together without merge conflicts from your visual tests.

How do you debug tests in Backstop?

“But it works on my machine.” If you build UIs for a living like us, you've probably heard this countless times. The differences in UI rendering between environments is a common source of frustration for developers. What you want from a visual testing tool is a standardized environment and the ability to debug in your own browser.

Backstop only tests with static images. It doesn’t allow you to inspect the rendered code that is under test. You have to run your test suite again locally to reproduce errors. This increases the time it takes to debug issues.

Chromatic prioritizes reproducibility. It doesn’t just capture an image snapshot, it archives the DOM, styling, and assets at the specific point in time of a test. You can inspect the code behind every change in your own browser via the Chromatic web app. This allows developers to debug exactly what users and teammates would see.

How do you collaborate in Backstop?

Backstop doesn’t come with collaboration features out of the box.

Chromatic was built for collaboration from the start. Product teams can review UI changes together in one place. Engineers, designers, and product managers can browse the test library online without needing to pull code, migrate, or maintain a dev environment. You can comment and assign reviewers to your visual tests.

What browsers does Backstop support?

Backstop pegs which browsers are available to the package version you have installed. By default it uses Chromium. New versions have new browsers while old versions have old browsers. It’s up to you to maintain the package version and deal with upgrading the browsers to new versions. If you want to add additional browsers like Firefox or Safari, you need to install another library.

Chromatic supports the latest stable versions of the world's most common browsers. You will be automatically migrated as we improve our infrastructure. This covers 95.93% of users. We believe this is the most pragmatic way to balance price, performance, and ease of use.

Why? Browser rendering engines used to be quite different. Each browser implemented CSS differently, which meant developers had to check that the UI rendered right in every browser.

Modern browsers interpret CSS and JavaScript similarly and update themselves automatically. This negates the value of having browser farms with myriad versions. Testing with the latest stable version of the most popular browsers gives you high coverage at a pragmatic price.

What does Backstop cost vs Chromatic?

BackstopJS is free and that's pretty appealing. But the price tag does not factor in the time required for configuration, ongoing maintenance, and support.

We reckon the hassle-free workflow boost you get from Chromatic is a fantastic value. Chromatic is trusted by thousands of companies to run millions of tests a day. The infrastructure is effectively able to run your entire test suite in parallel for the fastest test times.

What’s more, Chromatic’s TurboSnap can provide up to 85% faster test runs and on average 41% cost savings for the same coverage. It works by tracking code to test only what's changed.

Here’s a summary of how paid tools like Chromatic differ from open source tools.

How does Chromatic visual testing integrate with my tech stack?

Adding visual testing to the software development lifecycle requires integrations with existing developer tools, git providers, and collaboration tools. In a nutshell, below are the most important integrations to consider.

Chromatic integrates with all of the above. Our integrations are as native as possible. We map 1-to-1 concepts like Storybook stories and Playwright tests with visual tests. For example, every story is transformed into a visual test automatically. And every Playwright test for a user flow gets visually tested in its end state (and optionally along the way). All of this reduces test maintenance for you.

Chromatic is perfect for Storybook

We maintain Storybook, the industry-standard open source tool for frontend developers. With Chromatic, we created a way to automatically visual test everything you develop in Storybook.

Chromatic tests map 1-to-1 with your stories. Whenever you update a story your tests are synced. This means visual tests are painless to keep up to date. Storybook's own visual tests feature is powered by Chromatic.

Ready to try Chromatic?

Finding the right visual testing tool can be hard. We hope this guide makes it a bit clearer. Trying Chromatic is easy, setup your project in less than 90 seconds. No credit card required.