Intermittent failures in production are the worst kind of bug. They don’t happen every time. They don’t leave a clean stack trace. They mock your attempts to reproduce them in staging. And when they strike, they usually do it at 3 a.m. on a Saturday. I’m Felix Okonkwo, and I’ve spent more nights than I care to count staring at dashboards that flicker red then green for no obvious reason. This article is a direct, technical walkthrough of how I approach these problems—no fluff, no theory, just what works when the heat is on.
Why Intermittent Failures Are Different
A deterministic bug is a gift. You can reproduce it, isolate it, and fix it. An intermittent failure is a statistical event. It depends on timing, load, data patterns, or environmental conditions that align only occasionally. The failure rate might be 0.1% of requests, but when you serve a million requests a day, that’s a thousand angry users. The core challenge is that your normal debugging toolkit—breakpoints, step-through, local repro—is useless. You have to think in terms of signals, not symptoms.
Most engineers reach for logs first. That’s correct, but insufficient. Logs tell you what happened at a single point in time. Intermittent failures are often about what didn’t happen: a timeout that didn’t fire, a lock that wasn’t released, a cache entry that expired a millisecond too early. You need to reconstruct the system’s state across multiple services and time windows. This requires a different mindset.
Step 1: Define the Failure Signature Precisely
Before you grep a single log line, write down exactly what “intermittent” means in this case. Is it a 500 error on a specific endpoint? A spike in p99 latency? A dropped message from a queue? Quantify it: error rate over time, affected user IDs, geographic distribution, time-of-day patterns. If you can’t describe the failure in a single sentence with numbers, you’re not ready to debug.
For example: “Between 14:00 and 14:15 UTC, the /checkout API returned HTTP 502 for 0.3% of requests, exclusively for users in the us-east-1 region, and only when the request payload exceeded 8KB.” That’s a signature. Now you have a filter, not a haystack.
Use your monitoring tools to slice the data. Pull error rates by endpoint, by status code, by instance, by availability zone. Correlate with deployment timestamps, config changes, and traffic patterns. Often the “intermittent” label is just a lack of granularity. A failure that looks random at the service level might be perfectly consistent for a specific pod or a specific database shard.
Step 2: Instrument the Boundaries, Not the Internals
When a request fails intermittently, the root cause is rarely inside your application logic. It’s at the boundaries: network calls, database queries, message broker interactions, external API invocations. These are the points where timing matters most. Add detailed logging or metrics around every boundary call. Capture not just success/failure, but latency, payload size, connection reuse, and retry attempts.
I keep a standard checklist for boundary instrumentation:
- Outbound HTTP: Log target host, request duration, status code, and whether a connection was reused or newly established. Intermittent 502s often trace back to a load balancer that recycles idle connections while your app’s connection pool still thinks they’re alive.
- Database queries: Log the query plan hash, execution time, rows returned, and transaction isolation level. A query that runs in 2ms 99% of the time but spikes to 2s when a particular index falls out of the buffer cache is a classic intermittent culprit.
- Message brokers: Log publish confirmations, consumer lag, and redelivery counts. A message that’s silently redelivered after a consumer timeout can cause duplicate processing that fails on unique constraints.
- Cache operations: Log hit/miss, TTL at time of access, and serialization errors. A near-expiry cache entry that returns stale data under high concurrency is a race condition waiting to happen.
Step 3: Trace a Single Failed Request End-to-End
If you have distributed tracing (Jaeger, Zipkin, or a vendor solution), this is where it earns its keep. Find a trace ID for one failed request. Don’t look at aggregates—look at the exact span timeline. Compare it to a successful request with the same parameters. The difference is often a single span that took 50ms longer, or a span that’s missing entirely because a service dropped the trace context.
Without tracing, you can build a poor-man’s trace using correlation IDs. Generate a unique ID at the edge, pass it through every service via headers or message metadata, and log it at every boundary call. Then grep that ID across all your log streams. Reconstruct the timeline manually. It’s tedious, but it works.
Pay attention to the order of operations. In a successful request, the database query might complete before the cache update. In the failed request, the cache update might finish first due to a thread scheduling hiccup, causing a subsequent read to see stale data. These ordering anomalies are invisible in aggregate metrics.
Step 4: Stress the System Asymmetrically
You can’t reproduce the failure in staging because staging doesn’t have production’s traffic shape. Production traffic has bursts, slow clients, retransmits, and weird user agents that send double requests. To surface the bug, you need to stress the system in ways that mimic these patterns.
Here are techniques I use:
- Connection pool exhaustion: Artificially limit the pool size to 1 and blast concurrent requests. This exposes connection leaks and deadlock-prone code paths.
- Slow client simulation: Introduce a proxy that delays reading response bytes. This triggers write timeouts and reveals incomplete response handling.
- Clock skew injection: If your system uses timestamps for ordering (e.g., last-write-wins), skew the clocks on different nodes by a few seconds. This surfaces race conditions that depend on NTP sync.
- Data pattern replay: Capture a sample of production payloads that correlate with failures, then replay them at high volume. A specific JSON field that’s null 0.1% of the time can break deserialization if your code doesn’t handle it.
Run these tests in a pre-production environment that mirrors production as closely as possible—same instance types, same database version, same kernel parameters. The goal is not to replicate the exact failure, but to create conditions where the underlying defect becomes deterministic.
Step 5: Inspect the Garbage Collector and Runtime
Intermittent failures in managed-runtime languages (Java, .NET, Go, Node.js) often correlate with garbage collection pauses or runtime scheduler delays. A GC pause that lasts 200ms can cause a health check timeout, which triggers a pod restart, which drops in-flight requests. The failure looks like a network blip, but the root cause is memory pressure.
Enable GC logging with timestamps. Overlay GC pause times on your request latency graph. If latency spikes align with GC events, you have a memory problem. Common fixes: reduce allocation rate, tune heap size, or switch to a low-pause collector. For Go, look at goroutine scheduling delays using the execution tracer. A goroutine that’s starved for CPU because of a tight loop in another goroutine can delay request processing just enough to hit a deadline.
Step 6: Audit Your Retry and Timeout Logic
Ironically, the mechanisms designed to handle transient failures often cause intermittent failures. Retry storms, where a failed request is retried by multiple layers (client library, service mesh, application code), can amplify a small blip into a full outage. Timeouts that are too short turn normal latency variance into errors. Timeouts that are too long cause resource exhaustion as threads block waiting for dead backends.
Map every retry and timeout in your request path. For each one, ask:
- Is the timeout value based on measured p99 latency plus headroom, or is it a guess?
- Does the retry use exponential backoff with jitter, or does it hammer the backend at a fixed interval?
- Is there a retry budget (e.g., max 3 attempts per request) to prevent infinite loops?
- Are retries idempotent? A retried POST that creates a duplicate resource is a data corruption bug masquerading as a transient error.
I once spent two weeks chasing an intermittent “duplicate key” error in a database. The culprit was a client library that retried inserts on connection timeouts without checking if the original insert had succeeded. The fix was adding an idempotency key, not tweaking the database.
Step 7: Check the Infrastructure Layer
Application code gets the blame, but the infrastructure is often the silent accomplice. Load balancer health checks that flap between healthy and unhealthy cause requests to be routed to dying instances. Kubernetes readiness probes that don’t account for warm-up time kill pods before they’re ready to serve. Network overlays drop packets when MTU settings mismatch.
Correlate application errors with infrastructure events: pod restarts, node rebalancing, security group changes, TLS certificate rotations. A 0.1% error rate that spikes every 90 days might align with certificate renewal, where a few clients reject the new cert due to an outdated trust store. These are not code bugs, but they manifest as code bugs.
If you’re on a cloud provider, pull the infrastructure event logs for the affected time window. Look for maintenance notifications, host migrations, or network path changes. A brief network partition that isolates a few nodes can cause leader election chaos in a distributed system, resulting in split-brain writes that fail silently.
Step 8: Use Production Traffic for Testing (Carefully)
When all else fails, you need to experiment in production. This is not reckless if you do it with controls. The key is to minimize blast radius while maximizing signal.
Techniques I’ve used successfully:
- Traffic shadowing: Mirror a percentage of production traffic to a debug instance with extra logging. The shadow instance doesn’t affect real users, but it sees the same input patterns.
- Canary with debug mode: Deploy a single instance with verbose logging or a profiler attached, and route a small fraction of traffic to it. Monitor error rates closely and be ready to pull it out if performance degrades.
- Feature flag for detailed tracing: For authenticated users, enable a flag that logs full request/response bodies for a random sample of sessions. This captures the exact payload that triggers the bug, but requires strict data privacy controls.
These techniques are invasive. They add overhead and risk exposing sensitive data. Get sign-off from your security team, set strict sampling rates, and have a kill switch ready. But when the bug only appears under real user behavior—a specific sequence of clicks, a particular mobile carrier’s header injection—this is the only way to catch it.
Step 9: Write a Postmortem That Prevents Recurrence
Finding the bug is half the battle. The other half is making sure it doesn’t come back in a different form. A good postmortem for an intermittent failure doesn’t just document the fix—it documents the detection gap, the diagnostic path, and the systemic changes that will surface similar issues faster next time.
Your postmortem should answer:
- What monitoring alert would have caught this before users noticed? Add it.
- What log or metric was missing that would have shortened the investigation? Instrument it.
- What assumption about timing, ordering, or idempotency was violated? Codify it as a design review checklist item.
- What test—chaos experiment, load profile, data fuzzer—would have caught this in staging? Automate it.
Intermittent failures are a symptom of systems that have grown beyond their original assumptions. Every one you fix is an opportunity to harden the system against the next one. That’s the job. Don’t just patch the code—patch the process.
FAQ: Intermittent Failure Debugging
What’s the first thing I should check when an intermittent error appears?
Check for recent changes: deployments, config updates, infrastructure modifications. Even if the change seems unrelated, it often shifts timing or resource usage enough to surface a latent bug. If you have a change log, correlate the error’s first appearance with the change timeline. If you don’t have a change log, start one now—it’s the cheapest diagnostic tool you’ll ever build.
How do I debug an intermittent failure that leaves no logs?
If the failure is silent—requests simply disappear or return empty responses—look at the network layer. Capture packets on the affected hosts using tcpdump or a service mesh’s tap feature. Compare a successful and failed request at the packet level. Look for RST packets, TCP retransmissions, or TLS handshake failures. Silent failures are often network drops that occur before your application code runs.
Why do intermittent failures often get worse over time?
Because the underlying condition—a slow memory leak, a growing database table, an accumulating log file—worsens gradually until it crosses a threshold. What was a 0.01% failure rate becomes 0.1%, then 1%. Plot the error rate over a long time window (weeks, not hours). If it’s trending upward, you’re dealing with a resource exhaustion problem, not a random race condition. Look at memory, disk, file descriptors, and connection counts on the affected instances.
Can intermittent failures be caused by client-side behavior?
Absolutely. Clients that retry aggressively, send malformed headers, or close connections mid-request can trigger bugs in server-side code that assumes well-behaved clients. If the failure correlates with specific user agents, IP ranges, or geographic regions, instrument your edge layer to capture raw request data for those segments. You might find a mobile app that sends double requests due to a bug in its networking library, causing race conditions on your end.
Debugging intermittent failures is a discipline, not a talent. It’s about systematic signal collection, boundary analysis, and a refusal to accept “cannot reproduce” as an answer. The systems we build are deterministic at the hardware level; every failure has a cause. Your job is to find the right lens to see it.