There’s a special kind of frustration that comes with a bug that only shows up once in a blue moon. Not enough to set off every alarm, but just enough to make you dread checking Slack at 2 a.m. The real cost isn’t the bug itself—it’s the hours your team burns trying to corner something that refuses to be cornered. You stare at dashboards. You replay the same log lines. You mumble “works on my machine” and blame the network. Stop. If the failure happens once every ten thousand requests, it’s not random. It’s just waiting for a very specific set of conditions. Let’s talk about how to hunt it down.
“Intermittent” Is Just a Fancy Word for “I Haven’t Found the Trigger Yet”
Calling a failure intermittent is a confession, not a diagnosis. Somewhere in your stack, a precise combination of state, timing, and input is lining up just right to break things. Maybe it’s a race condition that only fires when two requests hit the same record within a 50-millisecond window. Maybe a garbage collection pause runs long because your heap is 87% full and a promotion fails. Maybe a load balancer health check and a real request collide on a thread pool that’s one thread short. The conditions are rare, but they’re repeatable. Your job is to shrink the haystack until the needle is obvious.
First, get specific. “The checkout endpoint throws a 500 sometimes” is a complaint, not a starting point. You need the exact error message, the full stack trace, the HTTP method, the endpoint path, the timestamp down to the millisecond, and any request payload that might have triggered it. If your logs don’t give you that, you’re flying blind. Fix the logging before you do anything else. You can’t debug a ghost.
Look at the Seams, Not Just the Fabric
Application code usually gets the logging love. The seams between systems? Not so much. Load balancer health checks, database connection pools, outbound HTTP calls, message queue consumers—these are where intermittent failures love to hide. A connection pool drains for 200 milliseconds and nobody notices until a request lands in that gap. A DNS lookup times out once because a resolver was restarting. A TLS handshake fails because an intermediate certificate expired, but only for clients that don’t cache the chain.
Structured logging at every boundary, with correlation IDs that survive across services. If a request dies, you should be able to follow its entire journey without stitching together timestamps by hand. Using distributed tracing? Check your sampling rate. A 1% sample is great for dashboards, but it’s a statistical guarantee you’ll miss the rare stuff. For debugging, crank it to 100% on the affected service. Yes, it’ll cost you in storage. Cheaper than another all-nighter.
Your Assumptions About the Stack Are Probably Wrong
Intermittent failures have a knack for exposing the lies we tell ourselves about how the stack works. I once lost three days to a file upload endpoint that failed 0.01% of the time. The error was a timeout, but only for files between 2.1 and 2.3 MB. The smoking gun: nginx had a default client_max_body_size of 2 MB. The backend accepted the connection anyway and then just… hung. The frontend load balancer had a 60-second timeout, the backend had 30, so the connection dangled until the client gave up. The fix took five minutes once we saw the interaction.
Here’s a checklist of assumptions to gut-check:
- Timeouts: Map every timeout in the request path—load balancer, reverse proxy, app server, database client. A 30-second app timeout sitting behind a 29-second proxy timeout creates a one-second window where the proxy kills the connection and the app never knows.
- Connection pooling: Are you leaking connections? A pool that slowly drains under a specific traffic shape will cause failures that vanish after a restart, only to creep back hours later.
- Serialization: Does your code assume a JSON field is always present? A malformed payload from a buggy client—missing a nested field, sending a string instead of an int—can trigger a deserialization path your normal traffic never exercises.
- Garbage collection: In managed runtimes, a full GC pause can outlast your health check timeout. The orchestrator marks the instance unhealthy, shifts traffic, and by the time you look, the instance is back to normal and you’re left scratching your head.
Hypothesis First, Data Second
Staring at logs and hoping a pattern jumps out is a recipe for wasted hours. Write down a specific guess about what’s happening, then design a query or experiment to prove it wrong. Suspect a race condition? Look for requests that overlap in time and touch the same resource—same user ID, same database row, same file path. Suspect resource exhaustion? Correlate failure timestamps with CPU steal, memory pressure, file descriptor counts, or thread pool saturation. Suspect a specific client version? Filter by user agent and see if the failure rate clusters.
Real example: a payment service kept throwing “duplicate transaction” errors, but only occasionally. Hypothesis: two requests with the same idempotency key arriving within milliseconds. Test: grep logs for idempotency keys that appeared more than once in a 100ms window. Result: a mobile client was firing a retry on a background thread without cancelling the original request. Both threads used the same key. The fix wasn’t on the server at all.
Reproduce It in Production (No, Really)
“Can’t reproduce in staging” is the most expensive phrase in operations. Staging doesn’t have production traffic, production data shapes, or production’s weird network topology. If you can’t trigger the bug in staging, take the fight to production—but do it safely. Route a fraction of traffic to an over-instrumented instance. Use internal-only debug endpoints that dump the full request context. Send synthetic requests that mimic the failing pattern, with headers and payloads copied from real failures.
One trick I lean on: deploy a canary with verbose diagnostics, then route only the suspicious traffic to it. If the failure correlates with users in a specific region or requests with a certain content type, isolate that slice. The canary will fail, but you’ll capture the entire state at the moment of failure—heap dumps, thread stacks, connection pool status, the works. Then kill the canary and go read the tea leaves.
Sometimes It’s the Hardware (Even in the Cloud)
The cloud abstracts hardware, but it doesn’t make hardware problems disappear. A noisy neighbor on a shared host steals CPU cycles, and you see it as intermittent latency spikes. A flaky network interface on a physical machine drops packets, and your retry logic masks it until it doesn’t. ECC memory errors are rare, but they happen. If you’ve ruled out every software cause, it’s time to look at the metal.
Check your cloud provider’s instance health metrics. Look for hypervisor events, live migrations, or hardware maintenance notifications that line up with your failure timestamps. On bare metal, dmesg is your friend—search for MCE (Machine Check Exception) errors. I once saw a team trace a 0.001% failure rate to a single rack where a top-of-rack switch was flapping a port. The switch logs had the evidence the whole time. Nobody had looked.
FAQ
Why do intermittent failures tend to get worse over time?
Because the underlying condition usually gets more frequent. A slow memory leak eventually fills the heap, GC pauses stretch out, and timeouts start piling up. A connection pool leak drains available connections until every request fails. A database table growing without proper indexes makes queries slower and slower until they slam into a timeout wall. The failure rate climbs as the system drifts toward a cliff.
How do I get management to approve more production logging?
Show them the bill for not doing it. Tally the engineering hours already burned, the customer complaints, the risk of a full-blown outage if the root cause stays hidden. Propose a temporary log verbosity bump with a hard expiration date—say, one week. Most managers will greenlight a week of noisy logs over a weekend of emergency pages.
What’s the first thing to check when an intermittent failure pops up?
What changed. Deployment history, config tweaks, dependency updates, infrastructure modifications. Even a tiny change—a new connection pool setting, a library patch version bump, a firewall rule adjustment—can introduce failures that only bite under specific conditions. If you have a change log, correlate failure timestamps with change timestamps. If you don’t have a change log, start one right now.
Can monitoring tools actually cause intermittent failures?
Absolutely. Health check endpoints that do real work—querying a database, calling an external service—add load that can tip a system over the edge. Monitoring agents that collect too many metrics can eat CPU and memory, causing resource contention. Even log frameworks can block when their buffer fills up, introducing latency that cascades into timeouts. Your observability stack is part of the system; treat it as a suspect.
How do I deal with intermittent failures caused by third-party services?
You can’t fix their code, but you can stop their failures from torching your system. Retries with exponential backoff and jitter. Circuit breakers that stop calling a failing service before the backlog takes down your own threads. Fallbacks—stale caches, default responses, graceful degradation. And instrument every external call so you can hand their support team a timestamp, a request ID, and a stack trace instead of a vague “your API is slow sometimes.”