Picking the right visual regression testing tool
If you’re looking to visual regression test your app UI you’ve probably encountered a lot of options.
From open source libraries to full-fledged products, there are plenty of choices that promise to catch UI bugs. It can be time-consuming to find the right solution for your project, so we’ve gathered some info to help you figure out which visual tool is right for you.
Read on for an overview of visual regression testing tools, including features, workflows, and pricing. We hope this helps you figure out the best testing solution for your project.
Website vs web app
Most visual regression testing tools are designed to test websites at the page-level. They’re made for marketing pages, blogs, and news sites.
The problem that app teams discover is that visual regression testing a multi-state web app is very different than testing a collection of static pages. Web apps generate countless UI permutations based on user input and context. Because managing all these permutations is hairy, modern view layers like React, Vue, and Angular exist to help you divide complex app UIs into simple modular components.
Unfortunately, testing web apps built with components using ancient page-level testing tools results in a flood of false positives that you have to dredge through to find real changes.
Paid tools like Applitools, Percy, Screener offer features for testing pages. But the features geared to test static sites often aren’t that useful for componentized apps. Open source tools like Puppeteer, BackstopJS, and Jest Image Snapshot are much tighter in scope so they don't cover the entire testing workflow that professional teams encounter.
We started Chromatic because we were tired of complicated review flows (or none at all), false positives, a bunch of features we didn’t need, painful integrations, endless maintenance, and test specs that polluted project repos.
Chromatic is centered around tracking and visual regression testing components -the basic building blocks of apps. Testing at the component level makes it easier to scope tests and identify regressions in web apps (just like unit tests help you pinpoint functional bugs).
What's more, testing at the component level is much faster than the alternatives. Chromatic's infrastructure is effectively capable of running all your tests in parallel, no matter how many builds you run, or stories you write. For instance, customers often test 2000+ permutations in less than a minute.
With Chromatic, each component in your app is tracked every commit so you can catch UI changes whenever you push code.
Paid visual regression testing tools vs open source libraries
It’s obvious, open source libraries are free. And free is pretty appealing. So what's the difference between paid visual regression testing tools and open source libraries?
Both paid and open source tools rely on the technique of visually comparing images to find differences, but that's where the similarities end.
-
Open source tools are much tighter in scope than paid professional tools. They are intended to solve specific challenges very well, but not the whole problem space.
-
Paid tools encapsulate the same basic functionality of their open source counterparts. They then layer work-saving features valuable to professional developers.
Here is a quick comparison using Chromatic, a paid visual regression testing tool.
| Paid (Chromatic) | Open source | |
|---|---|---|
| Performance | ✅ Fast: Tests run in parallel in the cloud so developers can get nearly instantaneous test results. | Slow: runs tests on the local development machine or requires extra work to run elsewhere. |
| Collaboration | ✅ Help teams review UI changes together. The online review process allows team members to review and confirm changes without pulling code. | Not scoped for multiple devs. Each dev must pull code and re-run tests to review. |
| CI & Git workflow | ✅ Seamless Git and CI workflow integration with notifications and PR statuses. | NA |
| Branches & merging | ✅ Supports working on many branches by tracking each branch separately. Intuitive workflows for merging UI changes. | Support varies |
| Reproductions | ✅ Easy reproductions because real code, styling, and assets for each component are indexed online. No Git required. | Checkout the branch and run the code to reproduce changes. |
| History | ✅ Tracks every component version so flipping through different versions is quick and easy. | NA |
| Storage | ✅ Snapshots are indexed online automatically. | Snapshots live in your repository (repo bloat). Or live in parallel which requires extra hosting and keeping them in sync. |
| Developer experience | ✅ Polished user flows and timesaving features | Barebones UX/UI |
| Maintenance | ✅ Round-the-clock support and frequent updates | Your team has to maintain |
| Deployment | ✅ A dedicated team constantly monitors and improves performance of the service | Your team has to manage deploys and downtime |
Which should you choose?
It depends on the project and the team. If you're a professional development team working on a critical app, the work-saving and collaborative features of a paid solution are a must-have. For instance, Chromatic pays for itself within days for small teams and hours for medium-sized organizations.
If you're the solo developer of a modest app or a seed-stage startup whose product is constantly changing then you may want to look at the open source options.
Development team vs individual developer
It takes a team to build and maintain a full-featured app. Like you, we understand that complexity grows and communication muddies the larger the team and project. Not everyone recognizes that visual testing is a team activity. You might ask a designer: “does this looks right?” or a QA person: “can you break this?”
There are plenty of choices but few are designed for the team workflow. Open source libraries like BackstopJS, Puppeteer, and Webdriver focus on the solo developer experience. They’re scoped for testing on the local machine but quickly become slow and blocking when used across a team.
Teams need collaboration
A tool like Chromatic is designed for collaboration from the start. Everyone can take part in the UI review process with the online merge manager. Multiple developers can work on multiple feature branches simultaneously. When it's time to merge, the process is painless.
Engineers, designers, and product managers can browse the component library online without needing to pull code, migrate, or maintain a dev environment. No more “this isn't working on my machine”.
How many components?
"What if I don't have that many components?"
A common concern with visual regression testing is that it's premature optimization. The thought process usually goes like this:
- There aren't many components so it's easy enough for developers to confirm them manually
- There isn't enough active development on components to warrant visual regression tests
It's easy to assume that visual regression testing is meant solely for established component libraries. Not true. A single component can realistically live in hundreds of places throughout your app. Making one tiny tweak will propagate to all the places the component is used. This is an enormous frontend risk!
Whether you have one or one hundred components makes no difference. When evaluating visual regression testing, first consider the number of times your components are reused not just the total component count.
Start testing UI components early
The benefit of testing UI components -even when you don't have many- is getting the assurance that your components will look as expected in all the places that they're used. Even if you rarely make changes, the changes you do make tend to have an enormous impact. The work-saving impacts of visual regression testing compound with the adoption and size of your component library.
Do you use a component explorer like Storybook?
Storybook is a component explorer that helps frontend teams ship UI components. As maintainers and sponsors of Storybook, we understand how integral Storybook is to app teams.
Chromatic's first-class integration with Storybook means developers can reuse their stories as visual regression tests without extra work writing new tests.
Every story is a test
Chromatic tests map 1-to-1 with your stories. Whenever you update a story your tests are synced. This means visual regression tests are painless to keep up to date.
Your Storybook is uploaded automatically
Every time you push code, the components in your Storybook are uploaded to Chromatic. This means everyone on your team can participate in reviewing. No pulling code required.
Reproductions
Chromatic doesn’t just capture an image of each component, it’s a living re-creation of component’s DOM, styling, and assets. Whenever you’re reviewing changes, you see high-fidelity components that are fully inspectable. This makes it painless to reproduce and confirm component changes because you’re seeing exactly what your users are.
No hassle branching and merging
Work on features together or separately. With branching and merging, you can make as many feature-branches as you need and seamlessly merge them back into production. It all happens behind the scenes, and just works. No configuration needed.
Get the free handbook
Want to learn more about visual regression testing? See why of visual regression testing is practiced by companies like Slack, Lonely Planet, and Walmart. Check out our free 31-page Visual Testing Handbook.
Ready to try Chromatic?
There are a lot of facets to consider with visual regression testing tools. We hope this guide makes finding the right tool a bit clearer.
Trying Chromatic is easy, install the addon and setup your project in less than a minute. Free plan. No credit card required. Sign up now.