You shipped the code. It passed every test. Staging gave you a clean bill of health. Then, at 3:00 AM, your pager screams. A service failed. By the time you log in, it’s working again. The logs show a timeout, a dropped connection, or a cryptic stack trace that vanishes on retry. Welcome to the special hell of intermittent production failures. These aren’t bugs; they’re ghosts. And you need to become a ghost hunter.
I’m Felix Okonkwo, and I’ve spent the better part of a decade chasing these phantoms across distributed systems. This isn’t a guide about positive thinking or “best practices” that only work in a vacuum. This is a technical, step-by-step approach to trapping a transient failure and dissecting it until it gives up its secrets.
1. Acknowledge the Physics of Distributed Systems
Before you touch a log file, you must accept a hard truth: your system is a distributed system. Even a monolithic application running on a single server is a distributed system when it talks to a database, a cache, or a message queue. The network is not reliable. Clocks are not synchronized. Garbage collection pauses exist. Your first step is to stop looking for a “bug in the code” and start looking for a violation of the laws of physics.
Intermittent failures are almost always a symptom of a timing issue, a resource exhaustion boundary, or a race condition that only manifests under specific load patterns. The code logic is often correct in isolation. The failure emerges from the interaction between components under stress. If you start by reading the code line-by-line, you will waste hours. Start by mapping the interaction.
2. Instrument Before You Investigate
If your system is already on fire, you need data. If you don’t have the data, you are blind. The first action in any intermittent failure scenario is to add targeted instrumentation. Do not just “turn on debug logging.” That’s a rookie move that will flood your storage and mask the signal with noise. You need surgical precision.
Focus on these four data points for every request that traverses the failing path:
- Latency histograms for every hop. A 99th percentile spike in a downstream call is the most common culprit. Standard metrics libraries (Prometheus, StatsD) aggregate this, but for intermittent issues, you need raw trace data.
- Connection pool states. Log the active, idle, and pending waiters for every connection pool at the moment of failure. A pool exhaustion event is often silent in application logs but screams in pool metrics.
- Garbage collection pauses. If you’re on a managed runtime, correlate failure timestamps with GC logs. A 200ms stop-the-world pause will blow through your timeout budgets.
- Kernel-level socket queues. Use
ss -tniornetstat -antpto check the Recv-Q and Send-Q. A non-zero Recv-Q means the application isn’t reading fast enough. A non-zero Send-Q means the remote isn’t acknowledging data.
3. The Timeout Cascade Is a Lie
Most developers set timeouts arbitrarily. A 30-second HTTP timeout, a 5-second database timeout, a 1-second cache timeout. When a failure occurs, they see a timeout exception and assume the downstream service was slow. That’s often wrong. The downstream service may have responded in 2 milliseconds, but the calling service’s thread pool was exhausted, so the request sat in a queue for 31 seconds before even being sent. The timeout you see is the caller’s timeout, not the callee’s latency.
To debug this, you need to compare the client-side elapsed time with the server-side processing time. If the server processed the request in 5ms but the client waited 5000ms, the problem is local resource saturation. Check thread pool sizes, connection pool limits, and circuit breaker states. A tripped circuit breaker that moves to half-open and then back to open can cause exactly this pattern. The fix is rarely “increase the timeout.” That just pushes the bottleneck downstream. The fix is to understand why your resources are saturated and address the root cause: slow database queries, a misconfigured connection pool, or a thundering herd on cache expiry.
4. Trace the Context, Not Just the Logs
Logs are linear. Reality is a directed acyclic graph of spans. If you are grepping log files for a correlation ID, you are doing it wrong. You need distributed tracing. If you don’t have it, implement it now. Jaeger and Zipkin are open-source options that take an afternoon to integrate. The critical piece is propagating a trace context across every RPC boundary. Without it, you cannot reconstruct the causal chain of a failed request.
When you have traces, look for the “missing span.” An intermittent failure often leaves a partial trace: you see the ingress request, a call to Service A, and then nothing. The trace just stops. This usually means the process crashed, was OOM-killed, or hit a timeout so severe that the span was never exported. Correlate the timestamp of the last recorded span with system logs, kernel logs, and orchestration events (e.g., Kubernetes pod restarts). The root cause is often a resource limit you forgot to set.
5. Reproduce with Chaos, Not with Hope
Waiting for the failure to happen again is a fool’s game. You must induce it. This is where chaos engineering stops being a buzzword and becomes a survival skill. You don’t need a complex platform. Start with tc (traffic control) to inject network latency and packet loss. Use stress-ng to starve the CPU or consume memory. Use iptables to drop a percentage of packets to a specific downstream dependency.
The goal is not to break production. The goal is to recreate the exact boundary condition in a staging environment that mirrors production topology. If you cannot reproduce the failure, you do not understand it. Increase the blast radius of your chaos experiments gradually. Start by adding 50ms of latency to calls to your database. Then 100ms. Then drop 1% of packets. The intermittent failure will become deterministic. Once it’s deterministic, you can attach a debugger, add more instrumentation, and dissect it.
6. Check the Plumbing: File Descriptors and Port Exhaustion
This is the silent killer of production systems. Your application makes outbound HTTP calls. Each one opens a socket. The operating system treats sockets as file descriptors. You have a limit. When you hit it, calls fail with “too many open files” or, more insidiously, connections hang in a TIME_WAIT state, exhausting the ephemeral port range. The failure is intermittent because it only happens under peak traffic.
Check /proc/sys/net/ipv4/ip_local_port_range and ulimit -n. Monitor netstat -an | grep TIME_WAIT | wc -l. If you see tens of thousands of TIME_WAIT sockets, your application is not reusing connections properly. HTTP keep-alive is your friend, but it must be configured correctly on both the client and server. The client must not hold connections longer than the server’s keep-alive timeout, or you’ll get silent connection resets. This is a classic race condition that produces intermittent 502 errors in reverse proxies.
7. The Database Connection Pool Trap
Your application server has a connection pool of 50. Your database is configured for a maximum of 100 connections. You run two instances of the application. Math says you’re safe. Reality says you’re not. Connection pools are not static. Under load, an instance might grab 45 connections. The other grabs 45. A new deployment rolls out, and the old instances linger for 30 seconds during a graceful shutdown. Suddenly, you have 135 connections trying to squeeze into a 100-connection limit. The database starts rejecting connections with a cryptic “too many clients” error. The application retries, making it worse.
This is a deterministic failure that looks intermittent because it depends on deployment timing and traffic spikes. The fix is to set the pool size to (max_connections - (superuser_reserved + replication_slots)) / number_of_instances with a safety margin. Also, set a connection timeout on the pool that is shorter than the query timeout. A connection waiting in a pool queue is not a failed query; it’s a resource starvation event.
8. The Cache Stampede and the Dogpile Effect
A cache key expires. A hundred concurrent requests notice the cache miss and all race to regenerate the value. This is a cache stampede. The database, already under load, gets hit with a hundred identical expensive queries. Some timeout. The application retries. The retries add more load. The system tips over.
This failure is intermittent because it only happens when a popular cache key expires during high traffic. The fix is not to increase the cache TTL. The fix is to implement a lock on cache regeneration. Only one request should be allowed to recompute the value; the others wait on that lock or serve stale data. If you’re using Redis, a simple SETNX with a short TTL acts as a mutex. If you’re not using a distributed lock, you’re gambling.
9. The Network Is Not Reliable, Not Even a Little Bit
TCP guarantees delivery or notification of failure. It does not guarantee timely delivery. A retransmission can take seconds. A misconfigured load balancer can drop idle connections silently. A switch can flip a single bit. Your application must handle these realities. Intermittent “read timeouts” are often caused by a load balancer closing an idle connection while your application pool thinks the connection is still alive. The application grabs the dead connection, writes a request, and waits for a response that will never come. The timeout fires. The application retries on a new connection, and it works. The failure looks random.
Configure your HTTP client to enable TCP keep-alives and set an aggressive idle connection eviction policy. Test your connection pool with a load balancer that has a shorter idle timeout than your application. You will see the failures immediately. This is not a network problem; it’s a configuration mismatch.
10. Observability Is Not Monitoring
Monitoring tells you the system is broken. Observability lets you ask arbitrary questions about the system’s internal state without deploying new code. For intermittent failures, you need observability. You need high-cardinality dimensions on your metrics: request ID, user ID, session ID, server instance, container version. When a failure occurs, you need to slice the data by these dimensions to find the pattern. Is the failure correlated with a specific server instance? A specific database replica? A specific client library version? Without high-cardinality data, you are guessing.
Structured logging is the poor person’s observability. If you can’t afford a tracing system, at least log in JSON. Every log line must include the trace ID, span ID, and relevant business identifiers. Then you can use jq and command-line tools to group, filter, and count. It’s not elegant, but it works when you’re desperate.
11. The Postmortem Is a Design Document
Once you find the root cause, the work is not done. The failure happened because the system allowed it to happen. A missing timeout, a missing circuit breaker, a missing bulkhead. These are design flaws. The postmortem should produce action items that are specific, technical, and testable. “Add a timeout” is not an action item. “Set the connection timeout on the inventory service HTTP client to 500ms, with a retry budget of 2 attempts, and verify behavior under a simulated 2-second network delay” is an action item.
FAQ
Why do intermittent failures often happen at 3:00 AM?
Because that’s when batch jobs run. Backup processes, ETL pipelines, log rotation, and database maintenance windows are typically scheduled during low-traffic hours. These jobs consume I/O bandwidth, CPU, and connection slots. Your application, still serving a trickle of traffic, suddenly contends with a resource-hungry batch process. Timeouts spike. The on-call engineer gets paged. Check your cron schedules and job orchestration timelines before blaming the code.
How do I debug a failure that happens once a month?
You cannot rely on real-time observation. You need persistent, structured logs with a long retention period. You also need to capture a snapshot of system metrics at the exact moment of failure. Set up a trigger: when the error rate for a specific endpoint exceeds a threshold, automatically collect thread dumps, heap histograms, connection pool states, and top-like output from all affected hosts. Store this forensic snapshot alongside the error logs. When the failure happens again, you will have a complete picture of the system state at that instant.
What’s the most overlooked cause of intermittent timeouts?
DNS resolution. Your application resolves a hostname, caches the IP, and reuses the connection. The DNS record changes, or the load balancer rotates the backend IP. Your cached connection is now pointing to a dead or decommissioned server. The next request fails with a connection timeout. The application retries, resolves the hostname again, gets the new IP, and succeeds. This is a classic intermittent failure. Set your HTTP client to respect DNS TTLs and evict connections when the TTL expires. Better yet, use a connection pool that supports asynchronous DNS resolution and health checking.
How do I convince management to invest in observability?
Stop using the word “observability.” It sounds like a vendor pitch. Calculate the cost of the last intermittent failure. How many engineer-hours were spent debugging? What was the revenue impact of the downtime? Present a concrete proposal: “Implementing distributed tracing with Jaeger will cost $X in infrastructure and Y engineer-weeks. Based on the last incident, which cost $Z, the payback period is N months.” Speak the language of the business. If the numbers don’t justify it, then the failure wasn’t expensive enough to care about. That’s a valid business decision, but it means you accept the risk.
Intermittent failures are not magic. They are the result of deterministic systems interacting at the edges of their design limits. Your job is to find that edge and either push it back or build a guardrail. There is no third option.