Flaky CI Tests From Live Third-Party APIs: Mock the Boundary, Don't Rerun

It is 4pm, the sprint ends tomorrow, and your CI build is red. Not because of your code. A test that calls the payment sandbox got rate-limited, or the SMS provider took four seconds to answer instead of two hundred milliseconds, or the third-party auth endpoint returned a slightly different JSON shape than yesterday. You hit "re-run job", it goes green, you merge. Everyone does this. Everyone knows it is a lie.

Here is the uncomfortable framing: a test that calls a live third-party API is not testing your code. It is testing your network, plus their uptime, plus their rate limiter, plus whatever their team shipped this morning, and then blaming your commit when any of those wobble. That is not a flaky test. That is a test pointed at the wrong thing.

The popular fix, wrapping the flaky test in automatic retries, makes the suite slower and less honest every week. The real fix is to stop the test from touching the external service at all, and to move the "does the real integration still work" question into its own small, separately scheduled suite where a failure means something.

First, separate the real root causes

"Flaky" is a symptom, not a diagnosis, and the fixes are completely different depending on the cause. Before you touch anything, put each flaky test in one of these buckets. Most teams have all five and treat them as one problem, which is why nothing improves.

Root cause	Tell-tale sign	The actual fix
Uncontrolled external API call	Fails at peak hours, on their outage, or intermittently with timeouts	Mock the boundary; move real checks to a scheduled suite
Async timing	Passes locally, fails in slower CI; a hardcoded sleep(2) nearby	Poll a readiness condition, do not sleep a fixed time
Shared mutable state	Fails only when run with others, passes alone	Fresh data per test in setup/teardown
Test order dependence	Fails when the suite order changes or runs in parallel	Remove reliance on a previous test's leftovers
Environment drift	Green locally, red in CI (or vice versa)	Pin versions, timezone, locale; run CI image locally

This article is about the first row, because it is the one people paper over with retries most often, and the one where the standard advice is actively wrong. But if your flakiness is really row two or three, mocking the network will not save you. Diagnose before you fix.

Why retry-and-rerun makes it worse, not better

Wrapping a flaky test in retries: 3 feels like progress. It is the opposite, for three concrete reasons.

First, it hides a real signal. When the payment provider genuinely goes down, your retry config swallows the failure as "just flakiness" and the build eventually passes on attempt three. The one time the external service is actually broken, the mechanism you built to ignore flakiness ignores the outage too. You have trained the suite to lie in exactly the situation where you needed the truth.

Second, it makes the suite slower for everyone, permanently. A test that sometimes needs three attempts triples its worst-case time, and CI minutes are not free, especially if you are paying for runners or waiting on a shared queue. Third, it removes the pressure to ever fix the test. A quarantined-but-retried test stays flaky forever because the retry makes it survivable. I go deeper on this specific trap in why CI auto-retry is hiding your flaky tests, not fixing them; the summary is that a retry is a painkiller you never stop taking.

The fix: mock at the boundary, verify the real thing separately

The principle is a clean split. Your normal test suite, the one that runs on every push and gates merges, must be deterministic and offline. It mocks the external service at the boundary of your own code. A separate, smaller suite verifies that the real external service still behaves the way your mock assumes, and it runs on a schedule, not on the critical path.

Step 1: Mock the boundary in unit and CI tests

Mock at the seam where your code calls out, the HTTP client or the provider SDK, not somewhere vague. In Python with requests, that means intercepting the call and returning a canned response, so the test is a pure function of its inputs:

import responses  # pip install responses

@responses.activate
def test_charge_marks_order_paid():
    # the boundary: pretend the provider returned a successful charge
    responses.add(
        responses.POST,
        "https://api.provider.test/v1/charges",
        json={"id": "ch_123", "status": "succeeded"},
        status=200,
    )

    order = create_order(amount=500)
    charge_order(order)          # this calls the provider internally

    assert order.status == "paid"
    # and assert you sent what you meant to send:
    assert responses.calls[0].request.headers["Idempotency-Key"]

This test now fails only when your logic is wrong. It cannot be rate-limited, it cannot time out, it runs the same on a plane over the Atlantic as in CI. Every language has this: nock in Node, WireMock in the JVM world, httptest in Go. Mock the transport, assert on both the outcome and the request you sent.

Step 2: Guard against the mock drifting from reality with a contract test

The obvious objection to mocking is real: your mock can lie. If the provider changes status: "succeeded" to state: "success", your green suite is now green about the wrong thing. That is what a contract test is for. It records one real response and asserts its shape, so when the provider changes the contract, exactly one test goes red and tells you to update your mocks. Where the sandbox itself is unreliable, record the real response once and replay it, a pattern I lay out in testing payment integrations when the sandbox itself is flaky.

Step 3: Replace fixed sleeps with polling on a readiness condition

A huge share of "async" flakiness is a sleep() that is long enough on a fast machine and too short in a loaded CI runner. Do not sleep a guessed duration; poll the actual condition until it is true or a timeout expires:

import time

def wait_until(predicate, timeout=5.0, interval=0.05):
    deadline = time.monotonic() + timeout
    while time.monotonic() < deadline:
        if predicate():
            return True
        time.sleep(interval)
    raise AssertionError("condition not met within timeout")

# instead of: time.sleep(2); assert job.done
wait_until(lambda: reload(job).status == "done")

This is faster in the common case (it returns the instant the condition holds) and reliable in the slow case (it waits up to the real timeout). A fixed sleep is the worst of both: slow always, and still flaky under load.

Step 4: Isolate state so tests do not depend on each other

If two tests share a fixture row and one mutates it, the pair passes or fails depending on order, which parallel CI will shuffle on you. Create the data each test needs in its own setup and tear it down after, or wrap each test in a transaction that rolls back. The rule is that any test must pass when run entirely alone. If it does not, it has a hidden dependency, and no amount of retrying will make that deterministic.

Verification: prove the flake is gone, do not assume

You cannot confirm a fix by watching one green run; flakiness is a probability. Run the specific test in a tight loop and require a perfect streak:

# pytest: run one test 200 times, stop at the first failure
pytest tests/test_charge.py::test_charge_marks_order_paid \
  --count 200 -x -q     # --count needs pytest-repeat

# any test runner: brute force it
for i in $(seq 1 200); do npm test -- charge || { echo "FAIL on $i"; break; }; done

Two hundred deterministic passes is evidence. Then track flake rate per test as a first-class metric going forward: how often each test fails then passes on the same commit. A test whose flake rate is above zero is on a list to be fixed or deleted, never silently retried. That single number, visible per test, is what keeps the suite honest over time.

What people get wrong

Global retry as a policy. Turning on retries for the whole suite means one genuinely broken test is invisible until it fails three times in a row, and a real outage is masked as noise. If you must retry anything, retry a named quarantine list, log every retry loudly, and treat the list as a bug backlog with an owner, not a permanent home.

"We need the real API or the test is meaningless." You need the real API in one place, on a schedule, watched by a human. You do not need it in the 300 tests that gate every merge. Conflating the two is what makes the whole suite hostage to someone else's uptime.

Mocking so much you test nothing. The opposite failure. If you mock your own database, your own serializers, and your own business logic, the test asserts that your mocks were configured the way you configured them. Mock at the process boundary, the network edge, and let your own code run for real inside it.

When it is still flaky

• It fails only in parallel. That is shared state or a shared port, not the network. Give each test its own database schema or its own randomized port, and stop reusing a single global fixture.
• It fails only in CI, never locally. Environment drift. Pin the timezone (TZ=UTC), the locale, the language runtime version, and run the exact CI container image locally until it reproduces. A test that assumes local time will flake across a timezone boundary forever.
• The mock passes but production breaks. Your contract drifted and no contract test caught it. Add or update the recorded-response check so a provider change turns exactly one test red with a clear message.
• A webhook or callback test is racy. You are asserting on an event that may arrive after the assertion. Make the receiver idempotent and poll for the recorded side effect instead of the inbound call, the same discipline as designing idempotent background jobs.

The mindset that ends flaky-CI misery is refusing to let a test's result depend on anything you do not control. Your merge gate should be offline, deterministic, and fast. The real world gets its own suite, on its own schedule, with its own alarm. Retrying is not a strategy; it is how a suite slowly stops meaning anything.