UI Tests

UI tests capture a visual snapshot of every story in a cloud browser environment. Whenever you push code, Chromatic generates a new set of snapshots and compares them against baselines. If there are visual changes, you verify if they’re intentional.

UI test

Enable

Enable visual tests for your project on the manage screen. All snapshots are taken in Chrome by default. This is also where you expand test coverage to additional browsers.

Enable UI Tests

Establish baselines

Once visual tests are enabled, you can establish baselines by running a Chromatic build in a new project or on a branch without an ancestor. This captures a snapshot of each story in a cloud browser and sets it as the baseline. Subsequent builds will generate new snapshots that are compared against existing baselines to detect UI changes.

View changes between baselines

Each build Chromatic compares new snapshots to existing baselines from previous builds. The list of changes are shown on the build page in the web app. The build will be marked “unreviewed” and the changes will be listed in the “Tests” table.

Build with unreviewed tests

What about component errors?

When a story fails to render it will be badged with “Component Error”. You will not be able to “pass” a build that has component errors. Fix story errors in Storybook and run tests again.

Verify UI changes

Chromatic detects UI changes but it’s still up to you to verify if changes are intentional. For intentional changes, you need to update the baseline so future tests will be compared to the latest baseline for the story. If a change is unintentional it needs to be fixed.

  • Accept change: This updates the story baseline. When a snapshot is accepted it won’t need to be re-accepted until it changes, even through git branches or merges.

  • Deny change: This marks the change as “denied” indicating a regression and immediately fails the build. You can deny multiple changes per build.

Snapshot that’s unreviewed

Tip: Speed up accept/deny using keyboard shortcuts and batch actions.
What about baselines in other branches?

Chromatic automatically changes the baseline snapshots that it uses for each build depending on your branch. Each branch has a separate set of baselines. This means you can update UI components on multiple feature branches in parallel without conflicts. Learn more »

How do I reproduce the snapshot?

Sometimes you need a closer look to determine why a snapshot is rendering as it does. Along with pixel and DOM diffs, Chromatic displays the interactive stories just as they appear in Storybook. Click on the snapshot image to go to the component screen. Then click on the “Canvas” tab to view the fully interactive story.

How are changes on builds different from those listed on the PR Screen 'UI Changes' tab?

UI tests (shown on the build screen) detect changes between builds, specifically, between the last accepted baseline and the latest build. This is useful for detecting defects during the development process and when merging to master to ship.

In contrast, the PR screen simply shows changes between the latest commit on the PR branch and the ‘merge base’. Think of it like the list of UI changes created by the code in the PR.

Merge

If you accept all the changes, the build will “pass”. Future builds whose stories have the same appearance will pass.

If you deny any of the changes, the build will “fail” and you will need to make code changes (and thus start a new build) to get the build to pass.

When your build is “passed” (all changes accepted), you’re ready to merge visual changes with confidence knowing that your UI is bug free. Chromatic will update the PR check for “UI Tests” to reflect the build status. W

After you merge your code, Chromatic will also apply accepted baselines to stories on the target branch ensuring you’ll only ever need to accept baselines a single time.

Build with reviewed tests

PR check for “UI Tests”

Chromatic adds a ‘UI Tests’ check within the status checks for your pull/merge requests. The badge shows errors or changes that need to be reviewed. Require the check in GitHub, GitLab, or Bitbucket to prevent accidental UI bugs from being merged.

PR for UI Tests

CI setup: Chromatic will return an exit code 0 on the CLI which can be used in CI/CD systems to indicate success and unblock deployment. Learn about CI.

Next: Learn about UI Review

Now that you can catch bugs during development, learn about how to invite stakeholders into Chromatic’s UI Review workflow to get the team sign off before merging.

Read next chapter


Keyboard shortcuts

Verify UI changes faster using keyboard shortcuts. Protip: Pressing 1 multiple times switches between the baseline and new snapshot in the 1up view. Keyboard shortcuts

False positives

It’s essential that your components and stories render in a consistent fashion between tests to prevent false positives. Some reasons your stories might not render consistently and ways you can avoid this include:

  • Randomness in stories: It’s not uncommon to use random number generators to generate data for complex component inputs. To avoid this, you can hard-code the input data, but often a more convenient solution is to use a tool like seedrandom which you can use to make your “random” number generator consistent.

  • Use of the current date/time: Dates and times are a testers bane! To get consistency in components or tests that use the current time, you can use a tool to also “seed” the time, like timemachine for the Date object.

  • Animations and video: Chromatic will attempt to pause all animations and video. However, you may need to configure Chromatic’s exact behaviour.

  • Unpredictable resource hosts: Resources that load from unpredictable or flaky sources may not load in time (15s) to capture. Workaround this by serving resources as static files in Storybook or using a placeholder service. Learn more about how we load resources.

  • Skip stories: Some stories may render unpredictably intentionally. If this is the case you may want to ignore the story from testing and move on.

  • Introduce a delay: As a last resort, you can try adding a delay. This will delay Chromatic’s snapshot for a specified amount of time. The trouble with this technique whilst it may make the problem less common, it may not eliminate it completely.

If you want to utilize the above techniques to ensure consistency for Chromatic’s tests, but you still want random elements for your local Storybook, you can use isChromatic() exported from our package to check for the Chromatic environment.

Troubleshooting

Build failed because of component errors

A build will fail if any of the snapshots fail to render (i.e. in rendering the latest version of the component, the snapshot throws a JavaScript exception). You’ll need to fix the code for errored components before we can pass the build.

Reviewing tests disabled on build screen

If a build isn’t the newest build on a branch, we disable reviewing the build; as any future builds will base themselves on the newest build, making approvals to this build pointless.

Note that in the case that there is a descendent build of this build on a different branch (for instance if the commit for this build was merged into that different branch), we do allow reviewing of this build. Future builds on this branch will use approved changes from the build; however future builds on the different branch will not—for this reason it is best to review builds before merging them.

"Didn't find any commits in this git repository in the last 100 builds."

This means that across the last 100 unique commits across all builds in your app, we didn’t find a single one that exists in the repository you ran this build against. Commits can go missing if you rebase or perform squash-merges, however, if all of the previous 100 builds’ commits are missing, it is likely something has gone wrong.

If you’ve reached this situation and can’t work out why, please let us know.

"Failed to find common ancestors with most recent builds within 1000 commits"

This means that although we found recent builds that were in your git repository history (see above), we couldn’t find any common history between your checked out build and those builds within 1000 commits.

Unless you are doing something unusual with your git repository, this is extremely unlikely. Either way, please let us know.

"Build X is based on a commit without ancestor builds."

When we create a build, we search your git history for a recent Chromatic build based on a commit that is an ancestor (i.e. a commit that is in the direct history of this commit). Unless this is the very first build, if we do not find one, we will show you this message.

This is typically unusual, because in order to run Chromatic on a commit, chances are the commit that added Chromatic to your app is an ancestor!

However, this situation can arise due to the following:

  1. You switched branches and re-ran Chromatic, without checking-in the code changes that installed Chromatic. In this case you can safely ignore this message.

  2. You rewrote history in merging the Chromatic installation code (e.g. using GitHub’s “Squash and Merge” or “Rebase and Merge” buttons). Learn how to resolve

  3. You are using a shallow clone of your repository when running Chromatic. Chromatic needs access to your full git history in order to find baselines (or at least the history until the previous Chromatic build, which depends on how often you push code/run builds). Learn about how we use Git for baselines »

  4. Something else happened—perhaps a bug at our end? Please let us know if this is the case.

Image size too large

We have a 25 million pixel size limit for image snapshots. This ensures fast and reliable performance for every build.

If your stories are larger than this, perhaps something has gone wrong? Let us know if you need this limit increased by chat or email.

Can I disable UI Tests if I prefer not to use them?

Yes. Go to the manage page for your project where you can disable UI Tests. Chromatic will no longer add status checks to your PRs for UI Tests once it is disabled.