The Problem With âIt Works on My Machineâ
Intermittent production failures are the worst kind of bug. They donât reproduce on demand. They skip staging entirely. They mock your unit tests. And when they finally hitâusually at 3 a.m. on a Saturdayâthey vanish before you can grab enough data to figure out what happened. Iâm Felix Okonkwo, and Iâve lost count of how many nights Iâve spent staring at dashboards that went red for six minutes and then cleared themselves. This article is a no-nonsense walkthrough of how to approach these failures, what instrumentation you actually need, and which patterns tend to hide in the gaps between your monitoring tools.
Too many teams treat intermittent failures as mysteries to be solved by intuition. Thatâs a dead end. These failures are deterministicâyou just donât have the data to see the determinism yet. The job is to shrink the observation gap until the cause becomes obvious. No guesswork. No superstition. Just systematic reduction of uncertainty.
Nail Down the Failure Signature First
Before you grep a single log file, define exactly what âintermittent failureâ means in this specific case. Vague descriptions like âthe API returns 500s sometimesâ are worthless. You need a tight signature: which endpoint, which HTTP method, which status code, which time window, which upstream dependencies were in play, and what the user actually saw. If you canât answer those, your first task isnât debuggingâitâs fixing your observability surface.
A real signature looks something like this: âBetween 02:00 and 02:10 UTC on weekdays, the /checkout POST endpoint returns HTTP 502 for roughly 0.3% of requests, correlated with a spike in p99 latency on the payment gateway sidecar.â Thatâs a signature you can work with. It gives you the time window, the frequency, the affected component, and a lead on an upstream dependency. Without that level of detail, youâre just flailing.
Instrument the Gap, Not the Symptom
Most teams respond to intermittent failures by adding more logging around the symptomâthe exact spot where the error surfaces. Thatâs backwards. The error is the effect. You need to instrument the causal chain upstream of the effect. If your API gateway is throwing 502s, the problem isnât in the gateway. Itâs in whatever the gateway calls that occasionally doesnât respond, or responds too slowly, or sends back a malformed payload that the gateway rejects.
Add structured logging with trace IDs at every network boundary. Not just HTTP callsâevery boundary. Database queries, gRPC calls, message queue publishes, file handle operations. The failure signature tells you which boundaries matter. If the 502s correlate with a specific upstream service, instrument the connection pool for that service: active connections, idle connections, wait time, connection timeouts, connection errors. One of those metrics will show a spike that lines up with the 502s. Thatâs your lead.
Time-Based Patterns Are Your First Real Clue
Intermittent failures that follow a schedule are almost always caused by a scheduled process. Cron jobs, batch processing, cache warming, log rotation, certificate renewal, connection pool recyclingâthese all run on timers. If your failure window is predictable, look for anything in your infrastructure that executes on a matching cadence. Donât limit yourself to your own code. Cloud provider maintenance events, database vacuum operations, Kubernetes node rotationsâthey all have schedules.
One technique I use over and over: overlay the failure timestamps with every known scheduled event across the stack. Infrastructure-as-code repos, CI/CD pipeline logs, cloud provider health dashboards, even internal team calendars. I once traced a 90-second spike in 504 errors to a Redis BGSAVE that ran every four hours. The fork latency was just long enough to starve the connection pool. The fix was a one-line config change to disable save-on-write and use AOF persistence instead. The investigation took three days. The fix took thirty seconds. That ratio is normal for this kind of work.
Correlation Isnât Causation, But Itâs a Damn Good Lead
When you donât have a time-based pattern, look for event-based correlation. Did a deployment happen shortly before the failure window? Did a dependent service experience a blip? Did a DNS TTL expire? Did a certificate rotate? Did a background job queue back up? Production systems are tightly coupled in ways architecture diagrams rarely capture. A memory leak in a logging sidecar can cause OOM kills that cascade into your application tier. A sudden burst of telemetry data can saturate a shared network link.
Build a timeline. Pull event logs from every system that touched the request path during the failure window. Include infrastructure events: pod restarts, node rebalancing, autoscaling actions, load balancer health check transitions. The failure is rarely where the error appears. Itâs usually one or two hops away, in a system you didnât think to check because âitâs always been fine.â
Reproduce by Amplifying the Stressor
If you canât reproduce the failure in a test environment, you havenât modeled the production conditions accurately. Intermittent failures are often triggered by resource contention that only appears under specific load patterns. Memory pressure, file descriptor exhaustion, thread pool saturation, connection pool depletionâthese donât happen at low traffic volumes. Your staging environment with 10 requests per second will never expose a race condition that only triggers at 10,000 requests per second with a specific interleaving.
Build a stress test that targets the suspected resource. If you suspect connection pool exhaustion, reduce the pool size in staging to a fraction of production and blast it with traffic. If you suspect a race condition in a shared data structure, increase concurrency far beyond normal levels. The goal is to amplify the stressor until the failure becomes reproducible on demand. Once itâs reproducible, itâs debuggable. Once itâs debuggable, itâs fixable.
Be careful with this approach. Amplifying a stressor can mask other failure modes. If you reduce the connection pool to 5 and the system fails, youâve proven that connection pool exhaustion can cause failureâbut you havenât proven that it did cause the production failure. You still need to verify that the production failure signature matches the amplified failure signature. Same error messages, same latency profile, same recovery behavior. If they match, youâve found your cause.
Distributed Tracing Is Not Optional
If youâre debugging intermittent failures in a microservice architecture without distributed tracing, youâre working blind. Logs tell you what happened inside a single service. Metrics tell you aggregate behavior. Only traces show you the end-to-end path of a specific failed request, including timing at each hop and the exact service that returned an error or timed out. This is not a nice-to-have. Itâs the difference between spending two weeks on a problem and spending two hours.
Implement tracing with a standard like OpenTelemetry. Propagate trace context across every service boundary. Sample aggressivelyâyou donât need 100% of traces, but you need enough to catch rare events. Tail-based sampling lets you keep all traces that contain errors or exceed a latency threshold while discarding healthy fast traces. This ensures you capture the failures without drowning in data. When the next intermittent failure hits, query your tracing backend for traces with http.status_code >= 500 during the failure window. The pattern will jump out.
Check Your Timeouts and Retries
A surprising number of intermittent failures are caused by timeout and retry configurations that work fine under normal conditions but break under degraded conditions. A service that normally responds in 50ms gets a 500ms timeout. When a dependent service slows to 400ms due to a cold cache, the callerâs 500ms timeout is still safe. But if the caller retries on timeout, and the dependent service is already struggling, the retry storm doubles the load and pushes latency past 500ms. Now every request times out, every request retries, and the system enters a death spiral.
Audit every timeout value in the critical path. Ensure retry budgets are enforced with exponential backoff and jitter. Check that circuit breakers are configured and actually trip when error rates spike. A circuit breaker that never opens is just decoration. Test your circuit breakers in staging by artificially slowing a dependency and verifying that the breaker opens within the expected error budget.
Kernel and Hardware-Level Causes
When youâve exhausted application-level explanations, go deeper. Intermittent failures can originate in the kernel or hardware. TCP retransmissions due to a flaky NIC. Memory errors that corrupt in-flight data before ECC catches them. Disk I/O latency spikes when a drive remaps a bad sector. CPU throttling from thermal issues. These are rare but real, and they leave fingerprints if you know where to look.
Check dmesg for hardware errors. Look at /proc/pressure/ for resource pressure stalls. Examine NIC error counters with ethtool -S. Correlate application error timestamps with kernel event timestamps. If you see a burst of TCP retransmissions that aligns with your 502 errors, youâve found a network problem, not an application problem. The fix might be a kernel parameter tuning, a driver update, or a physical hardware replacement.
Observability Gaps That Hide Intermittent Failures
Most production systems have blind spots where failures occur but no telemetry exists. Common gaps: the period between health checks, the startup sequence before logging initializes, the shutdown sequence after logging flushes, the internals of third-party libraries, the connection establishment phase before request logging begins. Intermittent failures love these gaps because theyâre invisible.
Map your observability coverage explicitly. For each component in the request path, identify what is logged, what is metered, and what is traced. Mark the gaps. Then fill them. Add startup logging that writes to a separate ring buffer. Add shutdown hooks that flush metrics before exit. Wrap third-party calls with thin instrumentation layers. The goal is to leave no gap wider than the duration of your shortest intermittent failure.
Postmortems That Actually Prevent Recurrence
Fixing the immediate cause is only half the job. The other half is ensuring you can detect and diagnose similar failures faster next time. A good postmortem for an intermittent failure doesnât just document what brokeâit documents what observability was missing, what assumptions were wrong, and what signals would have caught the problem earlier. Then it creates tickets to add those signals.
If you spent six hours correlating logs across four services to find a connection pool leak, the postmortem action item is: âAdd connection pool metrics (active, idle, pending) to service X and create a dashboard alert for pool saturation above 80%.â If you discovered that a cron job was the trigger, the action item is: âAdd cron job execution events to the central event log with start/end timestamps and exit codes.â Every painful investigation should make the next one faster.
FAQ
Q: How do I debug an intermittent failure that happens once a month?
A: You need persistent, long-retention telemetry. Standard log retention of 7 days wonât cut it. Set up a cold storage pipeline for traces and metrics with 90-day retention. When the failure occurs, youâll have the data to analyze it. Without that data, youâre waiting for it to happen again while you watchâand thatâs not debugging, thatâs hoping.
Q: What if the failure is in a third-party service I donât control?
A: Instrument your side of the boundary exhaustively. Log every request and response at the edge, including timestamps, latency, status codes, and response bodies if feasible. When the third-party service fails intermittently, youâll have evidence to present to their support team. Without that evidence, youâre filing a ticket that says âsometimes it doesnât workââand that ticket will go nowhere.
Q: How do I convince management to invest in better observability?
A: Track the cost of intermittent failures. Every hour your team spends debugging an issue that better telemetry would have caught in minutes is money lost. Every incident that impacts users is revenue and reputation lost. Present the numbers. Show the mean time to detect and mean time to resolve for recent intermittent failures. Compare that to what the numbers would be with the proposed instrumentation. Management responds to data, not technical arguments.
Q: Can machine learning or anomaly detection help?
A: Anomaly detection on metrics can surface intermittent failures faster than manual dashboard watching, but it wonât tell you the cause. Use it as a trigger for investigation, not a replacement. The real work is still in tracing the anomaly back to its source through systematic correlation and causal analysis.