Automatic flaky-test filtering

Flaky tests waste time by forcing you to investigate every failure: is it a real regression or just another unstable render?

Flake Filter answers that question automatically. It detects unstable tests and filters out their diffs so you can focus on real changes without false positives blocking your build.

It’s included in every plan and enabled by default. Read on to learn how it works.

Unstable tests shouldn't block your PR

When you run tests on an open PR, your goal is to identify actual regressions and not get bogged down debugging flaky tests that aren't even related to your code changes.

Unstable tests are usually caused by environmental factors such as an animation caught mid-frame, a slow-loading font, randomized UI data, or a network request that didn't finish in time. They trigger diffs intermittently, so it's hard to tell if it’s an actual regression without further investigation.

Flake filter auto-detects unstable tests

Chromatic renders each test multiple times to check whether it's stable. Tests that render unstably move into an ignored group, and their diffs do not block your build.

Unstable tests stay out of your way by default. You can always drill in to review the snapshots and diffs.

A couple of things to note:

• Multiple renders to check a test's stability still only counts as one billable snapshot.
• Chromatic re-evaluates every test for unstable behavior on every build. When a test stops flaking, it automatically re-enters the test suite.

Use traces to debug unstable tests

Ignoring an unstable test keeps your build moving, but instability is still a signal that the test needs attention. To help you act on it, Chromatic attaches a Playwright trace to every unstable test automatically. The trace captures the whole rendering session: network requests, console logs, and DOM snapshots.

Once you know the cause, the troubleshooting guide covers the common fixes.

Ignore tests that you're not ready to review yet

Along with Flake filter, we're also giving you the ability to ignore tests manually. Sometimes a stable test shows a change you're not ready to deal with, like a new story that isn't ready for review yet, or an unexpected diff after a rebase.

You can now ignore any test to get your build passing, then come back to the change later.

Green builds you can trust

Flaky tests put you in a hard spot, either dig into a change that has nothing to do with your code changes, or accept it without fully understanding why the change happened.

Chromatic now does that triage for you. Real diffs still block the build, but flaky ones get set aside. All unstable tests have attached traces to help you debug whenever you're ready to fix them.

Push a branch and see Flake filter in action yourself.