Intermittent Failures Are a Different Beast
Intermittent failures in production are the worst kind of problem. They don’t announce themselves. They don’t leave a neat stack trace you can grep for. They flicker in and out of existence, often tied to a specific confluence of load, timing, and state that you can’t easily recreate. If you’re the one on call, you know the drill: a spike in 500s, a flurry of confused Slack messages, and then—silence. The system heals itself before you can grab a thread dump. Hard crashes are almost comforting by comparison. At least they’re consistent.
This isn’t a problem you solve with intuition. It’s a problem you solve by turning the system inside out, instrumenting every layer, and refusing to trust anything you haven’t measured. The approach is part detective work, part experimental physics. You’re not just looking for a broken line of code. You’re looking for the exact set of conditions that make a working system suddenly fail.
Collect the Evidence Before You Form a Theory
When an alert fires, the natural impulse is to blame the last deployment or the database that’s been acting up. Fight that impulse. Intermittent failures are rarely monocausal. They’re emergent—a combination of factors that align just long enough to break something. If you jump to a conclusion, you’ll waste time chasing a ghost while the real trigger remains hidden. Start with the raw facts.
Define the failure in concrete terms. What exactly did the user see? A blank page? A timeout? A garbled response? Capture the timestamp, the affected endpoint, the user ID, the session token. If it’s a frontend issue, grab the browser console logs and the network waterfall for that session. If it’s a backend service, pull the request ID and trace it through every hop in your logging pipeline. You’re not just looking for the error. You’re reconstructing the sequence of events that led up to it. The failure itself is just the final frame of a bad movie.
You Can’t Debug What You Can’t See
If your production system isn’t instrumented, you’re debugging with a blindfold on. Structured logging is the bare minimum—every log line needs consistent fields: request ID, service name, latency, status code, user ID. Without those, you’re just grepping through unstructured text and hoping for a miracle. Metrics need to go beyond CPU and memory. You need application-level signals: thread pool saturation, connection pool wait times, queue depths, GC pause durations. These are the vital signs of a living system, and intermittent failures often show up as anomalies in these metrics long before they become user-facing errors.
Distributed tracing is non-negotiable if you’re running more than two services. A single request might touch half a dozen components. Without a trace, correlating a spike in Service D’s latency with a timeout in Service A is a guessing game. With a trace, it’s a five-second query. For intermittent failures, high-cardinality metrics are your best friend. Averages lie. p95 and p99 latencies tell the truth. Histograms can reveal bimodal distributions—a sure sign that two different code paths or resource pools are in play.
Correlation Is a Clue, Not a Conviction
Once you have a detailed timeline of the failure, start overlaying system-wide metrics. Look for anything that changed at the same time. Did CPU spike on one host? Did a cron job fire? Was there a network partition? Did a downstream service start returning slow responses? Dashboards with aligned time-series panels make this kind of visual correlation fast. If you don’t have one, build a dedicated incident-correlation dashboard that pulls in data from load balancers, app servers, databases, caches, and external dependencies.
Pay attention to leading indicators. A metric that shifts before the failure is far more interesting than one that shifts at the same time or after. A gradual climb in connection pool wait times that peaks right as errors appear? That’s a smoking gun for pool exhaustion. A sudden drop in cache hit rate followed by a spike in database latency? You’re probably looking at a cache failure or an eviction storm. The timeline matters as much as the values.
Reproduce the Conditions, Not Just the Request
You can’t reproduce an intermittent failure by replaying a single request in isolation. The bug depends on state—memory pressure, connection pool saturation, the timing of a competing thread. You need to recreate the environment that made the failure possible. That usually means load testing with realistic traffic patterns, injecting latency into dependencies, or squeezing resource limits.
If the failure correlates with high memory usage, run a canary instance with a smaller heap and watch the error rate. If it correlates with a slow upstream service, use Toxiproxy to inject artificial latency and see if the same failure mode appears. The point is to design a controlled experiment that triggers the failure on demand. Until you can do that, you don’t really understand the bug. You just have a hunch.
Get Your Hands Dirty in the Code
Once you have a strong correlation, it’s time to read the code paths involved. Look for shared mutable state, anything that isn’t thread-safe, and race conditions. Intermittent failures love concurrency bugs. They hide in lazy caching, stale data reads, and assumptions about the order of async operations. If the failure involves a timeout, trace every hop. A common pattern: Service A calls Service B with a 5-second timeout. Service B calls Service C with a 4-second timeout. Under normal load, everything finishes in 2 seconds. But when Service C slows down, Service B’s timeout fires at 4 seconds while Service A is still waiting. Service A gets a partial response or a generic error, and the real culprit—Service C—is completely invisible. Distributed tracing with span annotations makes this obvious in seconds.
Your Resilience Code Might Be the Problem
Here’s an uncomfortable truth: the code you wrote to make the system reliable often causes intermittent failures. Retry storms are the classic example. A request fails, triggers a retry, that retry fails, triggers more retries, and suddenly an already struggling service is buried under a self-inflicted DDoS. Circuit breakers that trip too eagerly can turn a transient blip into a hard failure. Fallback logic that returns stale or incomplete data can confuse downstream systems in ways that cascade. Audit your resilience patterns. Retries need exponential backoff and jitter—no exceptions. Circuit breakers need sane thresholds and reset timers. And your logging needs to capture every retry attempt: the original failure reason, the retry count, and the final outcome. Otherwise, you’ll see a success in the metrics and never know it took three attempts and nearly timed out.
Use Production Traffic as a Lab
Sometimes the only way to catch an intermittent bug is to watch it happen in production. That doesn’t mean attaching a debugger to a live server—please don’t do that. It means safely sampling or mirroring real traffic. Traffic shadowing copies a percentage of requests to a test instance that runs the same code but doesn’t touch real users. If the shadow instance logs the same errors, you can experiment with fixes without risking production. Another technique is incremental rollout with feature flags. If you suspect a recent code change, use a flag to shift traffic gradually—1%, then 5%, then 25%—while watching error rates. If the error rate tracks the rollout percentage, you’ve found your culprit. Roll back immediately and dissect the diff.
When the Bug Lives in the Infrastructure
Not every intermittent failure is a code bug. Hardware gets flaky. A switch drops packets under load. A disk develops slow sectors that cause I/O latency spikes. Cloud instances suffer from noisy neighbors that steal CPU or network bandwidth. These are harder to diagnose because you don’t control the physical layer. But you can still detect them by watching system-level metrics: CPU steal time, disk I/O await, network retransmits. If you see spikes that correlate with application errors, consider migrating workloads or enabling redundancy.
DNS is a silent killer. A stale DNS cache can route traffic to a decommissioned server. An intermittent resolution failure can cause random connection errors. Check your DNS TTLs and make sure your resolvers are healthy. Log DNS resolution times and errors at the application level. Don’t assume the infrastructure team has it covered.
Make Debugging a Team Sport
Debugging intermittent failures isn’t a solo activity. It requires a team that values observability and runs blameless postmortems. When an incident happens, document everything: the timeline, the hypotheses you tested, the data that confirmed or refuted them, and the final root cause. Share it with the team so everyone learns. Over time, you’ll build a playbook of common failure patterns and their signatures. Future investigations will go faster because you’ve seen the same movie before.
Invest in chaos engineering, but be deliberate about it. Start by injecting failures into a staging environment that mirrors production. Then, gradually move to production during low-traffic periods. The goal is to expose weaknesses before they become customer-facing incidents. If your system can’t handle a controlled experiment, it definitely can’t handle a real failure.
FAQ
Why do intermittent failures often correlate with deployments?
Deployments change the system’s state—new code, updated configs, restarted services. These changes can expose latent race conditions, shift timing assumptions, or increase resource usage just enough to push a component over its limit. The failure may not appear immediately because it needs a specific traffic pattern or a cumulative effect, like a slow memory leak, to trigger it.
How do I debug a failure that only happens once a week?
Set up long-term logging and metric retention. For rare events, you need weeks or months of data to spot patterns. Use anomaly detection on key metrics to automatically flag deviations. When the failure occurs, capture a full snapshot: thread dumps, heap dumps, network connection states, and recent request logs. Treat it like capturing a rare animal—you need traps set before it appears.
What’s the first thing to check when a service starts timing out intermittently?
Check connection pool utilization and wait times. Most intermittent timeouts come from exhausted connection pools—either to databases, caches, or downstream services. Look at the pool’s active connections, idle connections, and pending waiters. If waiters are queuing up, find out why connections aren’t being released. It’s often a slow query, a network blip, or a client that’s not closing connections properly.