Intermittent failures in production are the worst. They don’t show up in staging. They vanish when you try to reproduce them. And yet, at 3 a.m., they yank you out of sleep with a pager alert that clears itself before you’ve even found your glasses. If you work on systems that actually matter—payment processing, telemetry pipelines, authentication services—you know the drill. The error rate spikes to 2% for six minutes, then flatlines. The logs spit out a timeout, a dropped connection, or a stack trace from a library you didn’t write. Your first instinct is to blame the network. Don’t. Let’s walk through a methodical approach that finds the real cause instead of just restarting the service and hoping for the best.
Start With the Evidence You Already Have
Most teams skip this part. They jump straight to adding more logging or, worse, they start changing code. Before you touch anything, gather every piece of data from the affected time window. Pull application logs, infrastructure metrics, load balancer access logs, database slow query logs, and any distributed tracing data you have. Your job is to build a timeline of what the system was doing when things went sideways.
Look for correlation, not causation. Did the failure window line up with a deployment? A traffic spike? A cron job kicking off a batch process? A sudden drop in available database connections? I once spent two days chasing a bug that turned out to be a misconfigured backup script saturating a read replica’s disk I/O every Tuesday at 2:14 AM. The application logs screamed “connection timeout,” but the real culprit was a disk pegged at 100% for 30 seconds on a server nobody thought to monitor. The app was just the messenger.
Fix Your Observability Before You Fix the Bug
If you can’t answer basic questions about your system’s state during the failure window, your observability is broken. You need three things: metrics that show you what happened, logs that let you query specific events, and traces that connect requests across services. Most teams have metrics. Fewer have structured logs they can actually query. Almost nobody has tracing set up properly. Intermittent failures in distributed systems are nearly impossible to pin down without traces that show where a request spent its time and where it died.
For metrics, focus on the golden signals: latency, traffic, errors, and saturation. But don’t just stare at averages. Averages are liars. An intermittent failure affecting 1% of requests will be invisible in your P50 latency. You need percentiles—P95, P99, P999. If your P99 latency spikes while your P50 stays flat, you’ve got a tail latency problem. That’s your intermittent failure. Now you need to figure out what’s causing the tail.
Reproduce the Failure or Die Trying
Intermittent failures are intermittent because they depend on a specific set of conditions you haven’t identified yet. Your job is to find those conditions and recreate them. This is where most engineers give up and start guessing. Don’t be that engineer. Build a hypothesis and test it systematically.
Start with the simplest possible reproduction. If the failure involves a specific API endpoint, hammer that endpoint in a loop with varied payloads. If it involves a database query, run that query under load. If it involves a network call, simulate latency and packet loss. Tools like tc (traffic control) on Linux let you inject artificial delay, jitter, and packet loss onto network interfaces. A command like tc qdisc add dev eth0 root netem delay 100ms 20ms loss 1% can expose race conditions and timeout bugs that only surface when the network gets flaky.
If you can’t reproduce the failure in a test environment, you’re missing a variable. Check your production configuration. Are you using connection pooling? What are the timeout settings? Is there a circuit breaker that trips under certain conditions? I once burned three days trying to reproduce a connection reset error that only happened in production. The culprit was a 30-second idle timeout on a load balancer that didn’t match the 60-second keepalive on the application server. The load balancer was killing connections the app thought were still alive. A quick tcpdump on the production host caught the RST packet and solved the mystery in 20 minutes.
Use Production Traffic as Your Test Bed
Sometimes you can’t reproduce the failure outside of production because it depends on real traffic patterns, data shapes, or concurrency levels you can’t simulate. When that happens, you need to debug in production without breaking production. Feature flags, canary deployments, and traffic mirroring are your tools here. If you suspect a code change introduced the failure, roll it back with a feature flag and watch the error rate. If you suspect a performance regression, deploy a canary and compare its metrics to the stable version. If you need to test a fix, mirror a slice of production traffic to a test instance and see if the failure surfaces.
One technique I rely on heavily is adding targeted, high-signal logging to the failing code path. But don’t just log everything—that’s a fast track to drowning in noise and blowing up your log storage bill. Log the specific state that matters: the values of variables that control branching, the timing of critical sections, the exact error codes and messages from downstream dependencies. Use log sampling if the code path is hot. A 1% sample rate on a high-throughput endpoint will still give you hundreds of data points during a failure window.
Common Causes and How to Isolate Them
After debugging hundreds of these failures, I’ve found they usually fall into a few buckets. Here’s how to spot each one.
Resource Exhaustion
File descriptors, memory, threads, database connections—every resource in your system has a hard limit. When you hit it, requests fail. The failures are intermittent because the exhaustion is often transient: a slow memory leak triggers garbage collection, a connection pool drains and refills, a thread pool backs up under load and then recovers. Check your metrics for any resource creeping toward its limit. Look at the shape of the curve, not just the current value. A file descriptor count climbing steadily over days is a leak. A thread pool maxing out during traffic spikes needs tuning or backpressure.
Timeout Mismatches
This is the single most common cause of intermittent failures in distributed systems. Service A calls Service B with a 5-second timeout. Service B calls Service C with a 10-second timeout. Service C is slow, so Service B waits 10 seconds, but Service A has already given up and closed the connection. Service B then tries to write the response to a closed socket and gets an error. The fix is to make timeouts consistent and shorter as you go deeper into the call chain. Every service should have a shorter timeout than the service calling it. This is called timeout propagation, and if you don’t have it, you will have intermittent failures.
Race Conditions
Race conditions are the hardest to debug because they depend on timing. Two requests hit the same code path at the same time, and the interleaving of their operations causes a failure. These often lurk in caching logic, database updates, or shared mutable state. To find them, look for code that reads a value, modifies it, and writes it back without proper locking or atomic operations. Check your ORM for optimistic locking bugs. Check your cache invalidation logic. If you’re using a language with concurrency primitives, review every goroutine, thread, or async task that shares state.
Downstream Degradation
Your service is fine. The database, message queue, or third-party API you depend on is not. Intermittent failures from downstream services often look like your own failures because the error surfaces in your code. The key is to check the dependency’s metrics and status page during the failure window. If you don’t have access to those, instrument every outbound call with the same golden signals you use for your own service. Record the latency, error rate, and saturation of every dependency. When the failure happens, you’ll see a spike in dependency errors or latency that correlates exactly with your own error spike.
Build a Postmortem That Actually Prevents Recurrence
Once you’ve found the root cause, document it. But don’t write a postmortem that just describes what happened and says “we’ll add more monitoring.” That’s useless. A good postmortem identifies the specific condition that caused the failure, explains why your existing defenses didn’t catch it, and lists concrete actions that will prevent that specific class of failure from happening again. If the failure was caused by a timeout mismatch, the action item isn’t “add monitoring for timeouts.” It’s “audit all service-to-service timeouts and enforce a consistent timeout propagation policy.” If the failure was caused by a file descriptor leak, the action item is “add a linter rule that flags missing close() calls and set up alerts for file descriptor usage above 80%.”
Also, update your runbooks. The next engineer who gets paged for this failure shouldn’t have to repeat your investigation. Write down the exact commands you ran, the metrics you checked, and the log queries you used. Include the specific values that indicate the failure is happening. A good runbook doesn’t say “check the database.” It says “Run SHOW PROCESSLIST; and look for queries in ‘Sending data’ state for more than 5 seconds.”
FAQ
Why do intermittent failures often happen at the same time every day?
This usually points to a scheduled job or a traffic pattern. Check your cron jobs, batch processes, and any automated tasks that run on a schedule. Also check your traffic patterns—many systems have daily peaks that can trigger resource exhaustion or race conditions. A database backup that runs at 2 AM and saturates the disk is a classic example.
How do I debug an intermittent failure that I can’t reproduce?
First, improve your production observability. Add structured logging to the failing code path with enough context to understand the state when the failure occurs. Use distributed tracing to see the entire request flow. If the failure is rare, consider increasing your log sampling rate temporarily or adding conditional logging that only fires when the error condition is met. You can also use a circuit breaker to capture the request payload and state when the failure occurs, then replay that request in a test environment.
What’s the difference between an intermittent failure and a flaky test?
An intermittent failure happens in production and affects real users. A flaky test is a test that sometimes passes and sometimes fails without any code changes. Flaky tests are often a sign of the same underlying issues—race conditions, timeout mismatches, or resource contention—but they surface in your test suite instead of production. Fixing flaky tests with the same rigorous root-cause analysis will prevent those failures from reaching production.
Should I add retries to handle intermittent failures?
Retries can mask the symptom but they don’t fix the root cause. Worse, naive retries can amplify the problem by adding more load to an already struggling system. If you add retries, make sure they’re exponential with jitter, have a maximum retry count, and are idempotent. But your first priority should always be to find and fix the underlying cause. Retries are a bandage, not a cure.