Intermittent production failures are the absolute worst. They don’t fire often enough to trip alarms, but they happen just enough to bleed user trust and yank you out of bed at 3 a.m. You can’t reproduce them in staging. Logs look spotless. Metrics show a flatline. Yet somewhere in the stack, a request bombs, a queue stalls, or a database connection drops—and then everything snaps back to normal like nothing ever happened.
This is a practical walkthrough for engineers who are done guessing. I’ll lay out the common causes, the tools that actually help, and a systematic way to isolate these ghosts. No fluff. No theory that doesn’t map to a terminal window.
Why Intermittent Bugs Defy Standard Debugging
Most debugging workflows assume you can reproduce the problem on demand. You attach a debugger, set breakpoints, step through code, and find the faulty logic. Intermittent failures break that model because they depend on a specific confluence of state, timing, and environment that you can’t easily recreate.
These failures often stem from race conditions, resource exhaustion near limits, or silent corruption that only manifests under load. A thread pool that’s 99% full works fine until two requests hit the same millisecond. A DNS cache with a TTL of 300 seconds works until a downstream service migrates and your resolver returns stale records for exactly five minutes. A garbage collection pause that normally takes 2ms spikes to 200ms when the heap reaches a certain fragmentation pattern.
The common thread is that the system is operating within its designed parameters—until it isn’t. The failure is not a broken component; it’s a broken assumption about how components interact at the edges of their operating envelope.
Start with the Infrastructure Layer
Before you dig into application code, eliminate the physical and network layers. I’ve wasted days chasing a “code bug” that turned out to be a flaky NIC on a bare-metal host. The kernel logs had the evidence the whole time.
Check Kernel and System Logs
On Linux, dmesg -T and journalctl -k show kernel-level events. Look for:
- OOM killer invocations—even if your process survived, a sibling container might have been reaped, causing a brief cascade.
- Network link flaps or CRC errors on interfaces.
- Disk I/O errors or filesystem remounts to read-only.
- CPU throttling due to thermal limits or hypervisor contention.
If you’re in a cloud environment, pull the hypervisor metrics. AWS CloudWatch, GCP Operations Suite, or Azure Monitor can surface “steal time”—CPU cycles your vCPU wanted but the hypervisor gave to another tenant. A spike in steal time correlates directly with latency outliers and timeout failures.
Network Packet Loss and Latency
Intermittent network failures are insidious. A 0.1% packet loss rate is invisible in most monitoring dashboards but will cause TCP retransmissions that balloon response times for a small fraction of requests. Use mtr (My TraceRoute) between your application hosts and dependent services. Run it for hours, not minutes. Look for loss at any hop, especially the last one.
Check your load balancer logs. If you’re using AWS ELB, the surge_queue_length metric tells you if the balancer is spilling requests. A surge queue that’s non-zero for even a few seconds means some requests are being dropped or delayed. Similarly, check for connection resets in your reverse proxy (nginx, Envoy, HAProxy) logs.
Application-Level Patterns That Cause Intermittent Failures
Once infrastructure is cleared, the problem is in your code or its immediate dependencies. The following patterns are responsible for the majority of production-only, intermittent failures I’ve seen.
Connection Pool Exhaustion
Every database driver, HTTP client, and message broker library uses connection pools. When the pool is exhausted, new requests block or fail. The tricky part: pools drain and refill constantly, so the failure only appears when demand spikes align with slow backend responses.
Check your pool settings. A pool size of 10 with a connection timeout of 30 seconds works fine at 50 requests per second. At 500 requests per second, you’ll hit the limit. But you won’t see it in your average latency—only in your p99 and error rate. Monitor pool utilization directly. Most drivers expose metrics: HikariCP for JDBC, PoolSize for Npgsql, pool_size for Redis clients. Set alerts on utilization above 80%.
Thread Pool Starvation
Similar to connection pools, but harder to observe. When all threads in a fixed thread pool are blocked on I/O, new tasks queue up. If the queue is unbounded, latency spikes. If bounded, tasks are rejected. The failure is intermittent because it only happens when enough slow operations coincide.
For JVM applications, thread pool metrics are essential. Export them via Micrometer or JMX. Watch for rejected tasks and queue depth. In .NET, monitor ThreadPool.PendingWorkItemCount. In Go, goroutines are cheap, but you can still exhaust file descriptors or create contention in the scheduler. Use runtime.NumGoroutine() and debug.FreeOSMemory() judiciously.
Garbage Collection Pauses
GC pauses are the classic “everything looks fine except for these random spikes” culprit. A generational GC runs minor collections frequently and quickly. But when a major collection kicks in—especially with a large old generation—it can stop the world for hundreds of milliseconds. If your service has a p99 latency target of 200ms, a 500ms GC pause blows it apart.
Enable GC logging. For the JVM: -Xlog:gc*:file=gc.log:time,uptime,level,tags. For .NET, use COMPlus_GCLogEnabled=1. For Go, set GODEBUG=gctrace=1. Correlate GC pause timestamps with your latency spikes. If they align, you need to tune heap size, reduce allocation rate, or switch to a low-pause collector like ZGC or Shenandoah.
Cache Stampedes and Thundering Herds
When a popular cache key expires, dozens of requests simultaneously hit the backend to repopulate it. If the backend is slow, those requests pile up, causing timeouts. The failure disappears once the cache is warm again. This pattern repeats on every expiration cycle.
Fix this with probabilistic early recomputation (PER) or a locking mechanism on cache miss. In Redis, use SETNX to allow only one process to recompute. In application code, implement a single-flight pattern: multiple concurrent requests for the same resource share the result of the first one.
DNS and Service Discovery Staleness
Your application resolves a hostname to an IP and caches it. The downstream service scales in, changes IPs, or fails over. Your cached IP now points to a dead or overloaded instance. Most failures are transient because the cache eventually expires, but during the staleness window, a fraction of requests fail.
Check your DNS TTL settings. Many libraries default to caching indefinitely or for far too long. In Java, networkaddress.cache.ttl defaults to -1 (cache forever). Set it to a reasonable value like 30 seconds. Better yet, use a service mesh or client-side load balancer that actively health-checks endpoints.
Instrumentation: The Only Way to Catch Ghosts
You cannot debug intermittent failures with breakpoints. You need data from the moment of failure. That means structured logging, distributed tracing, and high-cardinality metrics.
Structured Logging with Context
Every log line must include a request ID, user ID, and session ID. Without these, you can’t correlate a single failed request across services. Use JSON logging so you can query fields directly in your log aggregator (Elasticsearch, Loki, Splunk).
Log at the boundaries: incoming request, outgoing request, error paths. Include the full error object, not just the message. A TimeoutException with a stack trace tells you where; a TimeoutException with the remote host, port, and elapsed time tells you why.
Distributed Tracing
Traces are the single most powerful tool for intermittent failures. A trace shows the exact path of a request through your system, with timing for each span. When a request fails, you can see which span caused it and what its inputs were. Jaeger, Zipkin, and OpenTelemetry are the standard options.
Instrument every RPC call, database query, and cache operation. Add baggage items for business context: customer tier, feature flags, experiment group. When you find a failed trace, you can filter for other traces with the same baggage to see if the failure is isolated or systemic.
High-Cardinality Metrics
Average latency hides intermittent failures. You need percentiles: p50, p95, p99, p999. But even p99 can be misleading if the failure is rare enough. Track the maximum latency over a rolling window. Track the count of errors by type, not just a generic error rate. A spike in ConnectionRefusedError tells a different story than a spike in SocketTimeoutException.
Use histograms. A Prometheus histogram with buckets at 1ms, 5ms, 10ms, 25ms, 50ms, 100ms, 250ms, 500ms, 1s, 2.5s, 5s, 10s lets you calculate any percentile and see distribution shifts. When your p50 is flat but your p999 is climbing, a histogram shows the tail growing before it breaches your SLO.
Reproduction Strategies When You Can’t Wait for the Next Failure
Sometimes you can’t afford to wait for the next occurrence. You need to provoke it.
Chaos Engineering with a Scalpel
Don’t randomly kill pods. That’s chaos monkey theater. Instead, inject specific failures that match your hypothesis. If you suspect connection pool exhaustion, reduce the pool size in a canary deployment and watch for errors. If you suspect GC pauses, allocate large byte arrays in a test endpoint to force a major collection. If you suspect DNS staleness, manually change a DNS record and observe your application’s behavior.
Tools like Gremlin, Chaos Mesh, or custom scripts with tc (traffic control) and iptables let you inject latency, packet loss, and DNS failures into a subset of traffic. Target a single instance, not the whole fleet, to limit blast radius.
Traffic Shadowing
Copy a percentage of production traffic to a staging environment that mirrors production infrastructure. This is expensive but effective. Use nginx’s mirror directive, Envoy’s request shadowing, or a custom proxy. The shadowed traffic hits real databases and services, so you need to ensure writes are idempotent or directed to a sandbox.
Compare latency distributions and error rates between production and shadow. If the shadow environment doesn’t exhibit the failure, the difference is in scale, configuration, or data shape. Narrow it down by making the shadow environment more production-like incrementally.
Replay from Logs
If you have structured request logs, you can replay the exact requests that failed. Tools like GoReplay or custom scripts can parse your access logs and resend requests to a test instance. This is especially useful for failures that depend on specific request payloads—malformed JSON, unusually large payloads, edge-case Unicode.
Case Study: The 2 a.m. Database Timeout
Let me walk through a real example. A payment service had intermittent 504 errors at roughly 2 a.m. every few days. The errors lasted 2-3 minutes, then vanished. The database team swore the DB was healthy. The network team saw no packet loss. Application logs showed SQLTimeoutException with no other context.
Step one: we added structured logging to the database client. Every query now logged the SQL text, bind parameters, and execution time. The next failure showed the queries were simple primary key lookups that normally took 2ms but were taking 30 seconds.
Step two: we correlated the timestamps with database server metrics. At exactly 2 a.m., disk I/O latency spiked from 1ms to 800ms. The DB was running a scheduled backup that did a filesystem snapshot. The snapshot froze I/O for a few seconds, which caused a queue of queries to build up. By the time the snapshot completed, the queue was so deep that some queries hit the 30-second client timeout.
The fix was trivial: move the backup window or increase the client timeout. But without the instrumentation, we would have spent weeks guessing.
Prevention: Design for Partial Failure
Intermittent failures are inevitable in distributed systems. The goal is not to eliminate them but to make them non-catastrophic.
Timeouts, Retries, and Circuit Breakers
Every external call needs a timeout. Not a default timeout—a consciously chosen timeout based on the SLO of the downstream service. If your payment gateway p99 is 500ms, set your timeout to 1s with a margin. Retry with backoff and jitter. But retries amplify load, so wrap them in a circuit breaker. After N consecutive failures, stop calling for a cooldown period. This prevents a slow downstream from taking down your entire service through resource exhaustion.
Graceful Degradation
When a dependency fails, don’t fail the entire request if you can return a degraded response. If the recommendation service is down, show default recommendations. If the analytics pipeline is slow, buffer events locally and flush later. Identify which features are critical and which are nice-to-have, and code them accordingly.
Idempotency Keys
Retries without idempotency cause duplicate operations. A payment retried due to a timeout can charge the customer twice. Use idempotency keys: a unique identifier sent by the client that the server uses to deduplicate requests. Stripe and other payment processors support this natively. Implement it in your own services for any mutating operation.
FAQ
What’s the first thing I should check when an intermittent failure appears?
Check the infrastructure layer: kernel logs, network packet loss, and hypervisor metrics. Eliminate physical causes before diving into application code. A surprising number of “code bugs” are actually a flaky NIC or a noisy neighbor on the hypervisor.
How do I convince management to invest in distributed tracing?
Show them the MTTR (mean time to resolution) for intermittent failures before and after tracing. Without traces, debugging a rare failure can take weeks. With traces, you can pinpoint the failing span in minutes. Translate that time difference into engineering hours and customer impact. A single major incident avoided pays for the tracing infrastructure.
Can I debug intermittent failures without reproducing them?
Yes, if you have sufficient telemetry. Structured logs with request IDs, distributed traces, and high-cardinality metrics let you reconstruct the failure from data. You won’t have a debugger, but you’ll have the exact sequence of events, timing, and inputs that led to the failure. That’s often enough to identify the root cause.
Why do intermittent failures often happen at the same time of day?
Because they’re triggered by scheduled events: backups, cron jobs, log rotation, cache expiration, or traffic patterns. Correlate failure timestamps with your system’s scheduled tasks. A backup that freezes I/O, a cron job that floods a queue, or a daily traffic spike that exhausts connection pools are common culprits.