Intermittent failures in production are the absolute worst. They don’t show up on your machine. They don’t appear in staging. They strike at 3 a.m., spike your error rate for ten minutes, then disappear without a trace. You’re left with a cryptic log line and a support ticket from a user who can’t reproduce the issue. If you’re Felix Okonkwo, you don’t wait for a pattern to emerge—you hunt it down with methodical, technical aggression.
This article cuts the fluff. We’ll walk through the root causes, the tools, and the mindset you need to catch these ghosts. No hand-holding, no theory without practice. Just the hard-won techniques that work when your pager goes off.
Why Intermittent Bugs Are Different
A deterministic bug is a logic error: given input X, you always get wrong output Y. Fix the code, deploy, done. Intermittent failures are probabilistic. They depend on state that’s hard to observe or control—timing quirks, resource exhaustion, network hiccups, even cosmic rays flipping a bit in RAM (rare, but real). Your unit tests won’t catch them because they don’t replicate the messy reality of production.
So stop staring at the code. You need data from the battlefield.
Step 1: Instrument Before You Need It
If you’re debugging an intermittent failure right now and your observability is garbage, you’re already behind. But you can still claw your way out. Add targeted instrumentation without a full redeploy if possible—feature flags, dynamic log levels, or attaching a debugger to a live process (carefully).
For the long game, your production system must emit structured logs, metrics, and traces. Not “maybe later.” Now. Every HTTP request, database query, and RPC call should be wrapped in a trace span. Logs must include correlation IDs. Metrics should track not just p99 latency but also the distribution of error codes, queue depths, and connection pool utilization. Without this, you’re blind.
Structured logging means JSON or something similar, not free-text strings. You need to query logs by fields: status_code:500 AND endpoint:/api/checkout. If you’re grepping plain text logs at 2 a.m., you’ve already lost.
Distributed tracing is non-negotiable for microservices. An intermittent timeout in Service A might be caused by a garbage collection pause in Service D. Without trace context propagating across calls, you’ll never connect the dots. Use OpenTelemetry. Instrument your HTTP client libraries, your database drivers, your message queues. If a library doesn’t support it, wrap it yourself.
Step 2: Characterize the Failure Signature
Before you touch any code, define the problem precisely. Intermittent failures often get lumped together as “flaky,” but that’s lazy. Break it down:
- Frequency: How often does it occur? 0.1% of requests? 5% during peak traffic? Only on Tuesdays?
- Duration: Does it self-resolve in seconds, or persist until a restart?
- Affected components: Is it one service, one endpoint, one database node, or everything behind a specific load balancer?
- User impact: Timeouts? 500 errors? Stale data? Silent data corruption?
- Correlated events: Deployments, config changes, traffic spikes, cron jobs, cloud provider maintenance windows.
Plot error rates against time, traffic, and deployment events. Overlay CPU, memory, and I/O metrics. Look for the moment the failure rate crosses your baseline. That inflection point is your first real clue.
Step 3: Common Root Causes (and How to Confirm Them)
Intermittent failures aren’t magic. They fall into predictable categories. Here’s how to test each hypothesis quickly.
1. Resource Starvation
Thread pools, connection pools, file descriptors, memory. When a pool is exhausted, requests queue or fail. The failure is intermittent because it only happens under concurrency that exceeds capacity.
How to confirm: Graph pool utilization (active, idle, pending) alongside error rate. If errors spike exactly when pending queue depth rises, you’ve found it. Common culprits: database connection pools with too-low max connections, HTTP client pools without proper timeout/eviction, or unbounded caches filling the heap.
Fix: Increase pool size, add circuit breakers, or—better—find why requests are slow and fix the root cause. A slow downstream service can exhaust your upstream pool even if the pool size is “technically” correct.
2. Race Conditions and Deadlocks
Two goroutines, threads, or processes accessing shared state without proper synchronization. The bug only manifests when the scheduler interleaves them just right. These are notoriously hard to reproduce, but production leaves clues.
How to confirm: Look for log entries that are out of order, or operations that completed faster than physically possible (e.g., a “write confirmed” before the “write started”). Enable lock contention metrics in your database. For application-level races, use ThreadSanitizer or Go’s race detector in a canary deployment that mirrors production traffic—not in staging, which never has the same concurrency patterns.
3. Timeouts and Retry Storms
A downstream service slows down. Your service times out, retries, and the retries overload the downstream, causing more timeouts. This positive feedback loop is a retry storm. The failure is intermittent because it only triggers when latency crosses the timeout threshold, which might be rare.
How to confirm: Trace a single user request that failed. Check if multiple backend calls were made for the same logical operation. Look for duplicate database records or duplicate external API calls. If you see a spike in “409 Conflict” responses, that’s retries colliding.
Fix: Implement idempotency keys. Use exponential backoff with jitter. Set a maximum retry budget per request. And for the love of sanity, don’t retry on non-idempotent operations without deduplication.
4. Garbage Collection Pauses
In managed-runtime languages (Java, Go, .NET, Node.js), a GC pause can cause request timeouts. The pause is intermittent because it depends on heap state and allocation patterns.
How to confirm: Enable GC logging. Correlate GC pause times with latency spikes. In Go, use GODEBUG=gctrace=1 and scrape the logs. In Java, use -Xlog:gc*. If p99 latency jumps during a GC pause, you’ve found it.
Fix: Reduce allocation rate, tune GC parameters, or switch to a low-pause collector (ZGC, Shenandoah). Sometimes the fix is as simple as reusing objects instead of creating them per request.
5. Network Blips and Partitioning
Packets get dropped. Switches fail over. Cloud provider networks have transient issues. Your system must handle these gracefully, but often it doesn’t.
How to confirm: Check TCP retransmit rates, network error counters (netstat -s), and cloud provider status history. Correlate with your error spikes. If you see SYN_SENT flood or connection refused errors, the network is suspect.
Fix: Implement proper connection pooling with health checks, fast failure detection (TCP keepalives, application-level heartbeats), and retry logic that respects idempotency. Use circuit breakers to isolate faulty dependencies.
6. Data Corruption or Inconsistent State
A database replica lags, a cache holds stale data, or a partial update leaves a record in an invalid state. The failure is intermittent because it depends on which node serves the request and the timing of replication.
How to confirm: Compare data returned from different replicas for the same query. Check cache hit rates and invalidation logs. Look for “phantom reads” or unexpected nulls in fields that should never be null.
Fix: Use strong consistency modes where correctness matters. Implement cache-aside with atomic invalidation. Add database constraints to prevent invalid state, not just application-level checks.
Step 4: Targeted Reproduction Techniques
Once you have a hypothesis, you need to reproduce the failure. But you can’t just “run it again” and hope. You need to amplify the suspected trigger.
Traffic mirroring: Capture production traffic and replay it against a canary instance with extra instrumentation. Tools like GoReplay or custom middleware can do this. The canary can have race detection, verbose logging, or experimental fixes.
Chaos engineering: If you suspect resource exhaustion, deliberately starve a test instance: limit CPU shares, cap memory, throttle I/O. If the error rate spikes, you’ve confirmed the mechanism. Use tools like tc (traffic control) to inject network latency and packet loss.
Deterministic simulation: For concurrency bugs, run the suspect code under a deterministic scheduler. This forces specific thread interleavings and makes the bug reproducible. It’s heavy lifting but sometimes the only way.
Step 5: The Fix Must Be Verifiable
Deploying a “maybe fix” and waiting to see if the error rate drops is amateur hour. You need a before/after comparison with statistical rigor. Define a clear metric (e.g., “p99 latency of /checkout endpoint”), collect a baseline over sufficient time, deploy the fix to a canary or a percentage of traffic, and compare distributions. Use a Kolmogorov-Smirnov test if you want to be formal, or just overlay histograms and eyeball the tail.
If the fix doesn’t change the metric, roll it back. Don’t leave dead code because “it might help somehow.” That’s how systems rot.
Step 6: Prevent Regressions
Once you’ve slain the dragon, make sure it stays dead.
- Add a regression test: If you can’t reproduce the exact failure in CI, test the mechanism. For a connection pool exhaustion bug, write a test that saturates the pool and verifies graceful degradation.
- Add a production monitor: Create an alert that fires when the conditions you identified (e.g., pool pending queue > threshold) are met, so you catch recurrence early.
- Document the incident: Not a bloated post-mortem, but a concise record: symptoms, root cause, fix, detection method. Link it from the code. The next engineer will thank you.
FAQ
What’s the first thing I should check when an intermittent failure alert fires?
Check deployment history and config changes. A huge fraction of intermittent failures are caused by a recent change that didn’t get properly validated under production load. If something was deployed in the last hour, roll it back first, then investigate. Don’t debug a moving target.
How do I debug a failure that only happens once a week?
You need to capture a full trace when it does happen. Set up conditional logging or tracing that triggers on the specific error signature and dumps the entire request context—headers, payload, downstream calls, timing. Treat each occurrence like a crime scene. Over time, patterns will emerge from the forensic data.
Is it ever acceptable to just add a retry and move on?
Only if you understand why the retry helps and you’ve confirmed the operation is idempotent. Blind retries mask symptoms and can cause retry storms. If you add a retry, also add a metric that counts retry attempts and alerts if the rate spikes. That way you’re buying time, not burying the problem.
What’s the most underrated tool for debugging intermittent failures?
Feature flags. They let you add debug logging, enable race detection, or test a fix on a small percentage of production traffic without a full deploy. If you don’t have a feature flag system, build one. It pays for itself the first time you isolate a heisenbug without redeploying at midnight.
Debugging intermittent failures is a discipline, not a lottery. Instrument ruthlessly, hypothesize from data, reproduce by amplifying triggers, and verify with metrics. Do that, and you’ll turn “flaky” into “fixed” faster than anyone expects.