Flaky CI is the kind of problem that quietly taxes an engineering team. A test suite that passes on your laptop, fails on the next push, then passes when you re-run the job does not just waste minutes. It erodes trust in the pipeline, slows merge cadence, and trains people to ignore red builds.
This post is a practical checklist for finding why a pipeline flakes and fixing the cause, not papering over it with retries. The goal is a build you can believe when it is green and act on when it is red.
What "flaky" usually means in practice
A flaky failure is one that is not stably reproducible from the same commit under the same inputs. Common shapes include:
Passes locally, fails in CI (or the reverse)
Fails randomly across identical jobs on the same SHA
Fails only under load, on certain runners, or at certain times of day
Fails after an unrelated dependency or infrastructure change
Treat flakiness as a product defect in your delivery system. If you cannot explain a red build, you cannot safely ship behind it.
Before you dig: capture the signal
Do not start by rewriting the test. First freeze evidence so you are not debugging folklore.
Pin the commit. Record the exact SHA, workflow, job, and attempt number.
Save artifacts. Logs, JUnit XML, screenshots, videos, coverage dumps, and container/OS versions matter more than memory.
Note the environment delta. Runner image, CPU count, memory, locale, timezone, network access, secret presence, and parallelism settings.
Count frequency. One failure in fifty is different from one in three. Track rate before you spend a day chasing a one-off.
If your CI platform supports it, keep failed-run logs for longer than successful ones. Flakes are intermittent by definition; lost logs turn hard problems into myths.
A working checklist for root cause
Work top-down. Most flakes fall into a small set of buckets. Check them in order of how often they show up in real teams.
1. Shared mutable state
Tests that write to the same database, filesystem path, cache key, port, or global singleton will collide when CI runs them in parallel. Symptoms: failures that vanish when you set workers=1, or that cluster around a subset of suites.
Fix pattern: isolate per test or per worker (unique DB schema, temp dirs, free ports). Prefer factories that clean up. Ban suite-level fixtures that mutate global config without restoring it.
2. Time and ordering assumptions
Hard-coded sleeps, reliance on wall-clock time, unsorted query results, and "eventually consistent" waits without a real condition are classic flake sources. CI runners are often slower or more contended than laptops, so a sleep that "usually works" locally becomes a roulette wheel in the cloud.
Fix pattern: wait on observable state (API ready, row present, file written) with a bounded timeout. Freeze time in unit tests. Assert on sets or sorted lists, not incidental order.
3. Race conditions in async and UI tests
End-to-end and integration tests often click or assert before the UI or service is ready. Locally you have a warm cache and a fast disk; in CI cold starts expose the race.
Fix pattern: explicit readiness checks, seeding that completes before the assertion, and fewer full-browser tests for logic that belongs in unit or API tests. If a UI test is the only coverage for a rule, that is a design smell, not a reason to keep a flake.
4. Resource limits and noisy neighbors
OOM kills, CPU throttling, disk full, and ephemeral runner contention look like "random" test failures. Check whether failures correlate with larger suites, matrix jobs, or shared self-hosted runners.
Fix pattern: raise limits where justified, split heavy jobs, cache dependencies correctly, and fail loudly on resource exhaustion instead of letting the process die mid-assertion.
5. External dependencies
Calls to third-party APIs, shared staging environments, package registries, or flaky containers introduce variance you do not control. A green local run with a mock and a red CI run against live staging is not mysterious; it is expected.
Fix pattern: mock or contract-test at the boundary for unit/integration suites. Reserve live dependency checks for a small, separately marked job that can fail without blocking every PR.
6. Non-hermetic builds
Floating dependency versions, latest tags, unpinned actions, and implicit reliance on preinstalled runner tools produce "same code, different result" behavior across days.
Fix pattern: lockfiles, digests for images, pinned action SHAs, and a documented runner baseline. Reproducibility is a flake-prevention feature.
Reproduce on purpose
Once you have a hypothesis, try to make the failure boring and repeatable:
Run the suspected test 50–100 times in CI and locally with the same seed and worker count
Increase parallelism deliberately to stress shared state
Run under a load tool or with CPU constraints if you suspect timing
Compare a failing job container image against a passing one byte-for-byte if you can
If you cannot reproduce after serious effort, quarantine with a ticket and a metrics watch—do not silently retry: 3 forever. Retries without a root-cause plan train the team to accept noise.
Fix, then prove it
A real fix changes the failure rate, not just the story. After the change:
Merge the fix behind the same job that used to flake
Watch the failure rate for several days of normal traffic, not one lucky green run
Remove temporary retries or quarantine flags once the rate is near zero
Add a guard if useful: a linter for sleeps in tests, a ban on shared temp paths, or a CI check that fails when flake quarantine grows
Document the incident briefly. The next engineer who sees a similar red build should find a short note: symptom, root cause, and the check that would have caught it earlier.
Process habits that keep pipelines trustworthy
Checklist discipline beats hero debugging:
Own flake budgets. Track flake rate per suite. Treat a rising rate like a sev for the delivery path.
Separate signal jobs from soak jobs. PR gates should be fast and hermetic; longer non-deterministic checks can run post-merge.
Prefer deleting bad tests over eternal quarantine. A test that cannot be made deterministic is not protecting you.
Keep local and CI close. Same major runtime, similar container, documented parity script. "Works on my machine" should be a rare exception, not the default narrative.
A short closing checklist you can paste into a ticket
SHA, job, attempt, and artifacts saved?
Failure rate measured (not guessed)?
Shared state, time/order, races, resources, externals, hermeticity checked in that order?
Reproduced under stress or ruled out with evidence?
Fix landed, rate re-measured, retries/quarantine removed?
Note written for the next person?
Flaky CI is rarely a mystery once you stop treating each red build as a one-off. Capture the signal, walk a fixed checklist, make the failure reproducible, and only then change the code. The payoff is a pipeline that means what it says—and a team that trusts it enough to move fast again.