A red pipeline is a symptom, not a diagnosis. The fastest engineers do not start reading logs top-to-bottom — they first place the failure in one of three layers: the application (your code / tests are genuinely broken), the pipeline infrastructure (registry, cache, credentials, dependency sources, deploy target), or the runner/agent (the machine executing the job — disk, memory, tool versions). Fixing the wrong layer is how teams end up re-running jobs for an hour. This guide is the triage.
The one question that localises 80% of failures
"Did this exact commit pass before?"
- A previously-green commit now fails, no code change → the environment changed. Infrastructure or runner. Suspect a moved dependency, an expired secret, a new base image, a full disk, a registry outage.
- Failure started exactly on the commit that touched the relevant code, and reproduces locally → application. Fix the code/test.
- Fails intermittently on the same commit → non-determinism: flaky test, race in parallel jobs, or a resource-starved runner. Do not "fix" it with a re-run.
Layer 1 — Application failures
These are legitimate and the pipeline is doing its job. Signals:
- Compile/type errors, unit or integration test assertions, lint/format gates, coverage thresholds.
- Reproduces locally with the same command the CI step runs.
- Localised to the build/test stage, tied to a specific diff.
What to check first: run the exact CI command locally (not your IDE's runner) with the same tool versions. Most "works on my machine" gaps are version drift — pin them (Step: Layer 2).
Layer 2 — Pipeline infrastructure failures
The code is fine; the plumbing broke. Signals and causes:
| Failing stage | Likely infrastructure cause |
|---|---|
| Checkout / clone | SCM auth token expired, LFS quota, network to the SCM |
| Dependency fetch (npm/pip/maven) | Registry down, a version yanked, a transitive dep moved, lockfile not honoured |
| Docker build | Base image tag changed under you, cache poisoned, build-arg/secret missing |
| Image push | Registry credentials expired, repository quota, immutable-tag conflict |
| Deploy | Cluster/target credentials rotated, changed manifest, environment gate, quota |
The tell for infrastructure is that it fails outside your source tree — before your code even runs, or when talking to an external system. The fix is usually pinning and hardening: pin base images by digest, honour lockfiles, rotate secrets with overlap, and make external fetches retry with backoff.
Layer 3 — Runner / agent failures
The most under-diagnosed layer, because the logs often look like an application error. The machine itself is the problem:
- Out of disk — accumulated Docker layers and build caches fill the agent; builds fail with "no space left on device" or cryptic tar/extract errors. Prune caches; size ephemeral runners.
- Out of memory — parallel jobs or a heavy compile/test OOM the agent; the job is killed (137) and the log just stops mid-step. Reduce parallelism or use a bigger runner class.
- Stale / poisoned cache — a corrupted dependency or Docker layer cache makes a good commit fail; a cache key that is too broad shares bad state across branches. Bust the cache key.
- Version drift — the runner image updated its default Node/Java/Python/Docker and your build assumed the old one. Pin the toolchain in the pipeline, not the runner image.
- Clock skew / TLS — a badly-timed agent breaks certificate validation on every HTTPS call.
The tell: multiple unrelated pipelines start failing at once, or the failure follows a specific runner/agent and clears on a different one. That is never your code.
Flaky pipelines: the "fixed by re-run" trap
A job that passes on re-run without any change is not fixed — it is flaky, and flakiness erodes trust in the whole pipeline until people reflexively re-run real failures too. Root-cause the flake:
- Test flakiness — order dependence, shared fixtures, real time/network in tests, unmocked external calls. Isolate and quarantine, then fix; don't let it mask regressions.
- Concurrency races — parallel jobs sharing a database, a port, a fixed resource name, or the same Terraform state lock (see the state-locking guide).
- Resource starvation — the runner was momentarily out of memory/disk; intermittent by nature.
A triage checklist
- Read the failing stage, not the whole log — it names the layer.
- Ask "did this commit pass before?" to split code-vs-environment.
- Check whether other pipelines are failing too (points to runner/infra).
- Reproduce the exact command locally with pinned versions.
- If intermittent, treat as flaky and root-cause — never accept a re-run as the fix.
Common wrong approaches
- Re-running until green. Hides flakiness and burns CI minutes; the failure returns.
- Editing application code to satisfy an infrastructure failure. e.g. loosening a test because the registry was down.
- Widening cache keys to "speed things up." Broad keys share poisoned caches across branches.
- Unpinned base images and toolchains. Guarantees future "green commit suddenly fails" incidents.
Related resources
- Terraform state locking and drift — a frequent source of pipeline concurrency failures.
- Kubernetes pod Pending decision tree — when the "runner" is a Kubernetes-based build pod that won't schedule.
- DevOps job support guide and SRE job support guide.
If a release pipeline is red and blocking a deploy right now, real-time proxy job support can help you localise the layer and unblock the release. And explaining a CI/CD failure clearly — which layer, why, how you'd prevent it — is exactly the kind of scenario a DevOps proxy interview support session prepares you to handle.
Last reviewed: September 2026.