Intermittent failures in production are the worst kind of bug. They don’t happen often enough to trip alarms, but when they do, you’re left with confused users, corrupted data, and a knot in your stomach. You can’t reproduce them on demand. Logs look clean—until they don’t. And the pressure to fix them comes from everywhere: management, customers, and your own sleep-deprived brain. I’ve spent years chasing these ghosts across distributed systems, embedded devices, and high-frequency trading platforms. Here’s what actually works.
Why Intermittent Bugs Are Different
Most developers treat intermittent failures like regular bugs with a low reproduction rate. That’s a mistake. A deterministic bug has a clear cause and effect. An intermittent failure is a system state collision—two or more conditions that only cause failure when they align perfectly. Think of it as a Venn diagram where the overlap is your outage. Your job is to map that overlap without being able to see it directly.
Common triggers? Race conditions that only manifest under specific load patterns. Memory corruption from a code path nobody hits. A cosmic-ray-induced bit flip in non-ECC hardware. Garbage collection pauses that just barely exceed a timeout threshold. DNS resolution delays that cascade only when combined with a retry storm. The list is endless, but the approach to finding them is consistent.
Step 1: Stop Guessing and Start Measuring
Your first instinct is to read code and theorize. Resist it. Code review has its place later, but right now you need data. Intermittent failures leave fingerprints—you just need the right instrumentation to see them.
Instrument the Boundaries
Every production system has boundaries: API calls, database queries, message queue operations, file I/O. These are where timeouts, retries, and partial failures pile up. Add detailed logging around every boundary with:
- Precise timestamps (millisecond resolution, minimum)
- Request IDs that propagate across services
- Latency measurements for each operation
- Return codes, even for successes—you need to know when something almost failed
Don’t log “database query completed.” Log “DB query ‘getUserProfile’ returned 1 row in 234ms, connection pool size 8/20.” The pool size matters because intermittent failures often correlate with resource exhaustion that doesn’t quite hit the limit.
Capture State Snapshots on Failure
When an error does occur, grab everything. Thread dumps, heap histograms, connection pool states, in-flight request counts, CPU utilization, garbage collection metrics. You’ll only get a few failures to analyze before the pressure mounts, so make each one count. Tools like jstack for JVM systems, GDB for native code, or language-specific profilers can be triggered automatically on specific error conditions.
Step 2: Build a Hypothesis from Patterns
Once you have rich failure data, look for correlations. This is detective work, not engineering. Plot failure timestamps against:
- Deployment events (even unrelated ones—infrastructure changes have side effects)
- Traffic patterns (failures often spike at 2 AM during backup windows, not at peak load)
- Upstream service latency (a 50ms slowdown in an auth service can trigger cascading timeouts)
- System clock adjustments (NTP slewing can break anything that compares timestamps)
I once tracked a “random” payment processing failure to a leap second insertion that caused a 1-second clock jump, which broke a monotonic time assumption in a locking library. The failure happened exactly once every 18 months. Without correlating timestamps to NTP events, we’d still be guessing.
Step 3: Force the Failure in a Controlled Environment
Reproducing intermittent bugs is about creating the conditions that make the failure probable, not about triggering the exact sequence. You need chaos engineering, but targeted.
If you suspect a race condition under load, don’t just run load tests—run them with network latency injection, CPU throttling, and clock skew. Tools like tc (traffic control) on Linux let you add 200ms of delay to specific ports. If the bug involves a database, run your test while a background process vacuums or reindexes. The goal is to widen the race window until the bug becomes reproducible on demand.
For memory corruption suspects, run under Valgrind or AddressSanitizer with a fuzzer hammering the inputs. Intermittent memory bugs often require a specific allocation pattern to overwrite the right bytes. A fuzzer combined with sanitizers can surface these in hours rather than months.
Step 4: Add Defensive Telemetry That Survives the Fix
Once you identify and patch the root cause, don’t rip out the instrumentation you added. Strip out the high-volume debug logs if they’re expensive, but keep the anomaly detectors. Add counters for the specific state combinations that caused the failure, and alert on them at low thresholds. The next intermittent bug will be different, but it will likely leave traces in the same boundary regions.
Implement distributed tracing if you haven’t already. OpenTelemetry or similar frameworks let you track a request across services and see exactly where time is spent. When a future intermittent failure occurs, you’ll have the timeline pre-built instead of reconstructing it from scattered logs.
Step 5: Write a Postmortem That Prevents Recurrence
A good postmortem doesn’t just document what happened—it identifies the systemic weakness that allowed the bug to reach production. Was the code review too focused on happy-path logic? Did the test suite lack timing variation? Was there no monitoring for the specific resource that became exhausted?
For intermittent failures, the postmortem should include a reproduction test that can be run in CI. If you can’t reproduce it deterministically, write a test that runs the suspect code path under randomized timing conditions for N iterations. A flaky test in CI is better than a flaky system in production—at least you’ll see it before users do.
Common Patterns and Their Fixes
Race Conditions in Asynchronous Code
You have two goroutines, threads, or async tasks that usually complete in order, but occasionally the second finishes first. The symptom is a null pointer, missing data, or incorrect state. The fix is explicit synchronization, but first verify the race exists. Add assertions that check ordering invariants. Run with race detectors enabled (Go’s -race flag, ThreadSanitizer for C++). If the race detector fires even once, you have your answer.
Resource Exhaustion Near Limits
Connection pools, thread pools, file descriptors, memory—all have limits. Intermittent failures happen when usage spikes just high enough to hit the limit, then immediately drops. Your monitoring shows 80% average utilization, but the 99th percentile touches 100%. Increase the limit or add backpressure. Better yet, graph the 99th percentile over time and alert when it exceeds 90% of capacity.
Timeouts and Retry Storms
A downstream service slows down slightly. Callers time out and retry. The retries add load, causing more timeouts. This positive feedback loop can turn a 50ms slowdown into a full outage. The fix: exponential backoff with jitter, circuit breakers, and request hedging (send the same request to multiple replicas and use the first response). But first, prove the storm exists by graphing retry rates against latency.
Garbage Collection Pauses
In managed runtimes, a GC pause can exceed your service’s SLA. The pause happens intermittently because it depends on allocation patterns and heap state. Enable GC logging with timestamps. Correlate GC pauses with latency spikes and errors. Tune the GC (e.g., switch to a low-pause collector like ZGC or Shenandoah) or reduce allocation pressure.
Clock Skew and Time Assumptions
Distributed systems often assume clocks are synchronized. They’re not. A node’s clock can drift seconds or even minutes before NTP corrects it. If your logic compares timestamps from different nodes, use logical clocks (Lamport timestamps, vector clocks) or tolerate skew explicitly. For absolute time, query a trusted time source rather than relying on local system time.
Tools That Earn Their Keep
Some tools are worth their weight in reduced debugging hours:
- Wireshark/tcpdump: When you suspect network-level issues, capture packets at both ends. A TCP retransmission or reset that happens only under specific congestion conditions is invisible to application logs.
- eBPF/BCC tools: Dynamic tracing without restarting processes. Trace kernel functions, syscalls, and user-space probes. Use
tcplifeto see short-lived connections that might be failing silently. - Chaos Monkey / Litmus: Not just for testing resilience—use them to reproduce intermittent failures by injecting the specific faults you suspect.
- AlloyDB/PostgreSQL audit logging: For database-related intermittents, log every query with parameters and timing. A specific query pattern might only cause a deadlock when run concurrently with a maintenance job.
FAQ
How do I debug an intermittent failure that happens once a month?
You can’t wait for the next occurrence. Set up conditional logging that triggers on the precursors to the failure, not the failure itself. If the bug involves a specific user action, log detailed state for every session that performs that action. When the failure eventually occurs, you’ll have the data leading up to it. Also, run accelerated simulations: if the failure happens once per million requests, generate ten million requests in a test environment with fault injection.
What if the failure leaves no trace in logs?
Then your logging is insufficient. Add logging at every decision point in the affected code path. If you don’t know the affected code path, add logging at every external interaction (network calls, disk writes, lock acquisitions). Use binary logging or ring buffers if volume is a concern. For crashes without stack traces, configure core dumps and ensure they’re captured even in containerized environments (set core_pattern in the host’s sysctl).
How do I convince management to give me time to fix an intermittent bug properly?
Quantify the impact. Calculate the error rate multiplied by affected users, then estimate revenue loss or support cost. Compare the cost of proper debugging (including instrumentation and chaos testing) against the cost of recurring incidents over a year. If the bug causes a 0.1% failure rate but affects a payment system processing $10M/day, that’s $10K/day in direct losses. Present the numbers, not the technical details. Management understands money.
Can I just restart the service and hope it goes away?
Restarting masks the symptom temporarily. If the bug is caused by slow resource accumulation (memory leak, connection leak, log file filling disk), restarting resets the counter and buys you time. But the bug will return, and the interval will shrink as load grows. Use the bought time to add instrumentation so you can catch it next cycle. Document the restart frequency and set an alert when it exceeds a threshold—that turns the restart itself into a signal.
Final Word
Intermittent failures are not magic. They are deterministic outcomes of specific, rare state combinations. Your job is to make those states visible and then make them impossible. Every hour spent guessing is an hour the bug stays in production. Instrument first, hypothesize second, reproduce third, fix fourth. In that order. Always.