CI Auto-Retry Is Hiding Your Flaky Tests, Not Fixing Them

A test goes red on CI. Someone hits re-run. It goes green. The pull request merges. Everyone moves on, and nobody writes down that it happened. Multiply that by a team and a year, and you have built a test suite that is green not because your code is correct but because your CI is configured to keep trying until the dice come up right.

Blanket retry-on-failure is the path of least resistance for keeping the pipeline green, and that is exactly the problem. It works. It is one config line. And it quietly deletes the single most valuable piece of information you had: the fact that, under some specific condition, your code or your test did the wrong thing.

The thesis of this article, stated plainly: automatic retries do not fix flaky tests, they launder them. The pain of a red build is the mechanism that would eventually force someone to fix the nondeterminism. Remove the pain and you remove the fix. The flake does not go away; it goes underground, and it takes a real regression with it the day one slips through.

Why the retry is worse than the red build

A failing test is a message. It says: under the exact conditions of this run, an assertion did not hold. The conditions matter as much as the failure. Which other tests ran first, what the parallel workers were doing, how loaded the runner was, what the clock said. A retry throws all of that away and replaces it with "eh, probably fine". You have converted a diagnosable event into a shrug.

And the incentive it creates is corrosive. On a team without retries, a flake is annoying enough that someone eventually gets fed up and fixes it, because the alternative is re-running builds by hand forever. Add automatic retries and that pressure vanishes. The test that flakes 5% of the time now passes on the second attempt 95% of those times, the build stays green, and nobody is ever annoyed enough to open the file. The flake becomes permanent precisely because it stopped hurting.

The truly dangerous case is the regression that looks like a flake. A genuine race condition you just introduced fails intermittently. With blanket retries, its first few failures are indistinguishable from your existing background flake, so CI retries them away and merges the bug. You will meet it again in production, where there is no re-run button.

Where flakes actually come from

Flakes cluster into two buckets, and the fix is different for each. Diagnosing which bucket you are in is the whole game.

Bucket	Cause	Tell
Nondeterminism in the test	Shared state leaking between tests, test-order dependence, unseeded randomness, real timing races (sleep-based waits, unawaited async)	Fails reproducibly when you fix the random seed and the test order; passes when isolated
Nondeterminism in the CI infrastructure	Parallelism exposing races, ephemeral runner variance, contention on a shared database, ports, or filesystem	Fails only under load or high parallelism; passes when run alone or serially on the same runner

The "works locally, fails in CI" complaint almost always lands in the second bucket, and it is not because CI runs different code. It is because your laptop is a quiet, fast, single-tenant machine and the CI runner is a constrained, heavily parallel, shared one. Your laptop never triggers the race because it never creates the contention. The CI runner does it on every run. The test is telling the truth; your development environment is too comfortable to hear it.

Step 1: Make retries visible before you make them safe

If you are going to retry at all, retry loudly. Most test runners can mark a test that passed only on retry, and CI can surface that as a distinct signal rather than a silent green. In pytest, for example, a rerun plugin reports reruns explicitly:

# pytest: allow one rerun, but the summary shows what was rerun
pytest --reruns 1 -p no:cacheprovider

# output distinguishes them:
#   1 passed, 1 rerun
# that "rerun" count is the number you must not ignore

The point is not the flag. The point is that a run which passed on the second attempt must look different from a run that passed on the first. If your dashboard shows both as a plain green check, you have no flake data at all. Emit the rerun count as a metric, per test, on every run.

Step 2: Quarantine, do not suppress

A test you know is flaky should not block merges, and it should not hide inside the main suite retrying itself either. Move it to a separate, non-blocking job. It still runs, you still collect its pass/fail data, but it cannot gate a deploy and it cannot mask the health of everything around it.

# pytest: mark the known-flaky test
import pytest

@pytest.mark.flaky_quarantine
def test_thing_that_races():
    ...

# CI: one blocking job for the healthy suite, one non-blocking for quarantine
jobs:
  tests:
    runs-on: ubuntu-latest
    steps:
      - run: pytest -m "not flaky_quarantine"   # gates the merge

  quarantine:
    runs-on: ubuntu-latest
    continue-on-error: true                       # visible, never blocks
    steps:
      - run: pytest -m "flaky_quarantine"

This is the crucial move. Quarantine preserves the signal for every other test: the main suite goes back to "green means green", so a genuine regression there is once again a real, trustworthy red. Meanwhile the flaky test keeps running where you can watch its rate, and it has a home that is neither "blocking everyone" nor "silently retried into invisibility". Quarantine must have an exit criterion, though, or it becomes a graveyard. Put a ticket and an owner on every quarantined test.

Step 3: Track flake rate per test

You cannot manage what you do not measure, and flake rate is the number that turns arguments into decisions. Record, per test, how often it fails across all runs over a rolling window. A minimal version is a job that parses the JUnit XML every run and appends to a store:

# after the test run, pytest can emit machine-readable results
pytest --junitxml=results.xml -m "not flaky_quarantine"
# ship results.xml to wherever you aggregate; compute failures / runs per test

With that in hand the policy writes itself:

• Flaked once, never again: noise. Leave it.
• Flakes on roughly 5% or more of runs: a real bug in the test or the system. Quarantine and root-cause.
• Flake rate rising over time: something is degrading. This is a signal, not a candidate for a higher retry count.

Step 4: Root-cause by forcing the condition

To fix a flake you have to make it fail on demand, which means recreating the condition the retry was hiding.

• Suspect test-order dependence? Run the suite in a randomised, fixed order and reproduce. pytest -p randomly or an explicit seed makes order deterministic so you can bisect which earlier test leaves state behind.
• Suspect unseeded randomness? Pin every source of randomness (language RNG, faker, UUIDs where feasible) to a seed printed in the output, so a failure is replayable.
• Suspect a timing race? Replace sleep-based waits with explicit condition polling, and run the test under artificial CPU load to widen the window the race lives in.
• Suspect infrastructure contention? Run the suite at higher parallelism than usual on a constrained machine. If it only fails at high worker counts, the shared resource, database, port, or temp directory, is your culprit.

Verification

You have actually fixed a flake, rather than moved it, when it fails zero times across a forced repeat run under the condition that used to break it:

# Run the previously flaky test 200 times in a row under the harsh condition
pytest -p randomly --count=200 tests/test_thing_that_races.py

Two hundred green in a row under randomised order and load is evidence. One green on a re-run is not. And at the suite level, the verification is a trend: your per-test flake dashboard should show the fixed test drop to a zero flake rate and stay there, while the quarantine job shrinks as tests graduate back into the blocking suite. If the number is not going down, you are still laundering.

What people get wrong

Raising the retry count when the flake rate climbs. This is the reflex the whole article is against. A rising flake rate is your system telling you the nondeterminism is getting worse. Answering it with --reruns 3 instead of --reruns 1 is turning up the music to drown out the smoke alarm. Every increment buys a little more green and a little less signal, and it compounds until a genuine regression sails through on attempt three.

Deleting or skipping the flaky test. The equal and opposite mistake. A test flakes because it exercises a real race or a real shared-state bug; deleting it removes the only detector you had for a defect that still exists. Quarantine is not deletion. The test keeps running and keeps reporting; it just stops blocking while you fix the cause.

Blaming "CI being CI". "It only fails in CI, CI is flaky" is a story that lets everyone off the hook. CI is not flaky; CI is constrained and parallel, and those constraints are exposing a real race that your comfortable laptop hides. The runner is doing you a favour by finding it before your users do. Treat "fails only in CI" as a diagnosis pointing at timing and contention, not as an excuse.

When it is still broken

1. Reproduce CI's constraints locally. Run the suite in a container with capped CPU and memory and the same parallelism CI uses. Many "unreproducible" flakes reproduce instantly once your machine is as starved as the runner. Docker makes this straightforward; if that workflow is new to you, my practical Docker on-ramp gets you there.
2. Isolate shared external state. A shared test database or message broker is the most common infrastructure flake source. Give each parallel worker its own schema or namespace so they cannot collide.
3. Check for time and timezone assumptions. Tests that depend on the wall clock, on ordering by timestamp, or on the runner's timezone flake when runs cross a second boundary or run in a different region. Freeze time in tests that care about it.
4. Look for the flake in the code under test, not the test. Sometimes the test is correct and the flake is a real concurrency bug in production code that only manifests under load. That is the best possible outcome of taking flakes seriously: the test did its job.

Keeping CI green is not the goal. Keeping CI honest is the goal, and a green build that was retried into existence is not honest, it is a build that has stopped telling you the truth about your code. Make the retries visible, quarantine what you cannot fix yet, measure the flake rate, and fix the nondeterminism at its source. A test suite you can trust is worth more than a pipeline that is always green.