Intermittent failures are the worst kind of production problem. They don’t announce themselves with a clear stack trace. They don’t leave a neat, reproducible set of steps. Instead, you get a trickle of user complaints, a spike in your error rate that vanishes before you can pull the logs, or a service that works perfectly until 3:00 a.m. on a Tuesday. In environments where infrastructure is a given—redundant power, ample bandwidth, and a fresh instance just a click away—you might shrug and spin up a replacement. In the systems I work with, often running in West Africa, that’s not an option. You might have one physical server in a locked room with a failing air conditioner, a satellite link that drops when it rains, and no spare parts for 200 kilometers. An intermittent failure here isn’t a curiosity; it’s a threat to a clinic’s patient records or a microfinance bank’s daily settlements. This article is about how to hunt down those ghosts when you can’t just throw more hardware at the problem.
What Makes a Failure “Intermittent” and Why It’s So Dangerous
An intermittent failure is a fault that appears, disappears, and reappears without an obvious, consistent trigger. It’s not a hard disk that has completely failed and sits there with a solid red light. It’s a disk that throws a single I/O error every 47 hours, causing a database transaction to abort, but then works perfectly for the next two days. In a high-availability cluster in a London data center, you might just replace the disk and move on. In a rural clinic in northern Nigeria, that single disk might be the only copy of the patient database, and the replacement budget is six months away.
These failures often stem from a combination of factors: a component that is marginally within spec, an environmental condition that fluctuates, or a software race condition that only triggers under a specific, rare load pattern. The challenge is that standard monitoring often misses them. Your CPU graph looks fine on a five-minute average, but a one-second spike to 100% caused by a backup script colliding with a report generation is invisible. The real work is in setting up the right traps to catch the ghost.
First, Rule Out the Physical World
Before you spend days instrumenting code, look at the physical layer. In many of the environments I deal with, the root cause is not a software bug but a physical constraint that the software doesn’t handle gracefully.
Power Quality Is Not Just “On” or “Off”
Mains power in many regions is unstable. Voltage can sag well below 200V on a nominally 230V line, especially when a nearby industrial motor kicks in. Most server power supplies can handle a wide range, but the cheaper switches, routers, or the inverter in a solar-hybrid system might not. A momentary voltage dip can cause a network switch to reboot, dropping packets for 30 seconds while it renegotiates spanning tree. To the application, this looks like a random database connection timeout.
What to do: Log power events independently. A simple Arduino-based monitor with a voltage sensor and an SD card can record sags and spikes with a timestamp. Correlate these timestamps with your application error logs. I’ve seen a case where a clinic’s server would fail every afternoon—turned out the cleaner plugged a heavy-duty floor polisher into the same circuit as the server rack, causing a voltage drop that made the UPS switch to battery momentarily, which in turn caused a brief network flap on a switch that had a failing capacitor.
Temperature and the Slow Creep of Death
Heat is a silent killer of electronics, but it rarely kills instantly. Instead, it pushes components to the edge of their operating envelope. A CPU might throttle, slowing down a critical thread just enough to cause a timeout. A hard disk might have a slightly increased seek error rate that the firmware corrects, but the added latency cascades into an application-level failure. In a server room with unreliable cooling—think a split-unit AC that freezes up and needs to be reset manually—the temperature can cycle between 20°C and 40°C daily. Your system might only fail at the peak.
What to do: Don’t rely on the server’s internal sensors alone; they can be inaccurate or only polled infrequently. Place a standalone temperature logger (even a cheap USB model) in the rack. Graph the temperature over a week and overlay it with your error timestamps. I’ve used a simple script that reads a USB thermometer and appends to a CSV file every minute. The correlation often jumps out immediately.
Network Intermittency: It’s Not Always the ISP
When a remote site loses connectivity, the first instinct is to blame the internet service provider. In many cases, that’s fair—microwave links fade in heavy rain, fiber cuts are common where construction is uncoordinated, and 4G towers get congested. But I’ve also seen plenty of failures caused by internal network issues that masquerade as ISP problems.
ARP Table Exhaustion on Cheap Routers
In a network with many devices—think a hospital with dozens of IoT sensors, workstations, and phones—a low-end router might have a limited ARP table. When the table fills up, the router stops resolving IP-to-MAC addresses for new or refreshed entries. Devices that were communicating fine suddenly can’t reach each other until an entry times out. The failure is intermittent because it depends on the number of active devices at any moment.
Fix: Check the router’s ARP table size and compare it to the number of devices on the subnet. A managed switch with a larger table or segmenting the network into VLANs can solve this. I’ve also seen this happen with cheap Wi-Fi access points used in point-of-sale systems; upgrading to an AP that supports 802.1Q VLANs and proper client isolation eliminated the problem.
DNS Timeouts and the Single Point of Failure
Many systems in low-resource environments use a single DNS server, often the ISP’s default. When that server is slow or unreachable, every name resolution can take seconds, causing application timeouts. The failure is intermittent because the DNS server itself might be overloaded at peak times. I’ve seen a district education office where teachers couldn’t upload exam results reliably. The culprit was a DNS forwarder on an old Windows Server that would stop responding for 10-15 seconds at a time under load. Adding a secondary DNS server (even a Raspberry Pi running dnsmasq) and configuring clients with a short timeout and retry solved it.
For a deeper look at building resilient network services on a shoestring, see my piece on Designing a Low-Cost, High-Availability Network for Rural Clinics.
Software Traps for Transient Ghosts
Once you’ve ruled out the physical and network layers, you’re left with the software stack. Intermittent software failures are often the result of race conditions, resource leaks, or time-dependent logic. The key is to instrument the system to capture the state at the exact moment of failure, without overwhelming the system with logging overhead.
Logging with Context, Not Volume
In a production system with limited disk I/O, verbose logging can itself cause intermittent failures by saturating the storage. Instead of logging everything, implement a circular buffer in memory that captures detailed trace information for the last N seconds. When an error condition is detected, dump the buffer to persistent storage. This gives you a high-resolution snapshot of the system state leading up to the failure without the constant I/O penalty.
I’ve used this approach on a Python-based data synchronization service running on a Raspberry Pi. The Pi’s SD card was slow, so continuous debug logging caused write spikes that delayed the main loop, which in turn caused timeouts. A ring buffer of the last 1,000 log entries, flushed only on error, allowed us to catch a subtle bug where a thread was being starved of CPU during a specific cron job.
Health-Check Endpoints That Reveal Internal State
A simple “/health” endpoint that returns 200 OK is almost useless for debugging. Instead, expose a detailed status endpoint (protected by authentication, of course) that shows internal queue lengths, connection pool status, last error timestamps, and resource usage. This allows you to poll the system during an incident and see what’s degrading before it fails completely.
For a Node.js application handling USSD payments, I added an endpoint that returned the current number of pending transactions, the database connection pool state, and the time since the last successful mobile money API call. When failures occurred, we could see that the connection pool was exhausted because the mobile money API was slow, not because our application was leaking connections. The fix was to add a circuit breaker that stopped accepting new requests when the pool was near capacity, rather than letting them queue and time out.
Reproducing the Unreproducible
You can’t fix what you can’t reproduce. But in a production system, you can’t just run a loop of test cases and hope to trigger the bug. You need to create a safe, controlled environment that mimics the production conditions closely enough to surface the failure.
Traffic Shadowing with Real Data
If you have a spare server (even an old desktop), set it up as a shadow instance. Mirror a copy of production traffic to it, but have it respond to no one. You can then experiment on this shadow system—restart services, inject delays, simulate resource constraints—without affecting users. This is especially useful for intermittent failures that seem to depend on specific data patterns. I’ve used tc (traffic control) on Linux to simulate network latency and packet loss, and stress-ng to simulate CPU and memory pressure, all on a shadow server cloned from the production image.
Time-Shifted Replay
For failures that occur at specific times—like month-end report generation—capture the incoming requests and replay them against a test system at a different time. Tools like gor (GoReplay) can capture HTTP traffic, but even a simple script that replays web server access logs with curl can work. The key is to replay the exact sequence and timing of requests that led to the failure. I once debugged a payroll system that crashed only when processing the last day of the month because a daylight-saving time calculation caused a thread to spin indefinitely. Replaying the access log against a test instance with the system clock set to the failure time reproduced it reliably.
When the Fix Is a Workaround
In an ideal world, you find the root cause and fix it permanently. In the real world, you might not have the source code, the vendor might have gone out of business, or the fix might require a hardware upgrade that’s not in the budget. A pragmatic workaround is often the only viable path.
The Watchdog That Actually Works
A watchdog timer that reboots the system when a service hangs is a blunt instrument, but sometimes it’s the right tool. The key is to make it smart enough to avoid rebooting during a legitimate long-running operation. For a Java application that would occasionally deadlock, I wrote a small external watchdog in C that monitored not just a heartbeat file, but also the application’s thread dump. If the heartbeat stopped and the thread dump showed all threads in BLOCKED state, it killed and restarted the JVM. This kept the system available while we negotiated a support contract with the vendor to fix the underlying deadlock.
Graceful Degradation Over Perfect Uptime
Sometimes, the best you can do is ensure that when a component fails intermittently, the system as a whole degrades gracefully rather than crashing. For a logistics tracking system that relied on an unreliable GPS module, I changed the software to cache the last known good location and serve that if the GPS was unresponsive for less than 60 seconds. The user saw a slightly stale position, but the application didn’t throw an error. This is a pattern I call “last known good”—it’s not perfect, but it keeps the system useful while you work on the root cause.
Building a Culture of Debugging, Not Blame
Intermittent failures can erode trust in a system and in the team that maintains it. Users start to see the system as unreliable, and management may pressure the technical team for quick fixes that make things worse. The only way out is a methodical, blameless approach to debugging.
Document every incident, no matter how small. Record the time, the symptoms, the environmental conditions, and what was done. Over time, patterns emerge. I keep a simple shared spreadsheet for each site I support, with columns for date, time, observed behavior, and actions taken. After a few months, it becomes clear that the “random” database errors always happen on Tuesday afternoons when the generator test runs, or that the “intermittent” network drops correlate with heavy rain. This turns a ghost into a manageable risk.
In resource-constrained environments, you can’t always fix the root cause. But you can understand it, plan for it, and design your system to survive it. That’s the difference between a system that’s fragile and one that’s resilient.
Frequently Asked Questions
How do I know if an intermittent failure is hardware or software?
Start by isolating the layers. If the failure correlates with environmental changes (temperature, power, time of day), suspect hardware. Run hardware diagnostics—memtest86 for memory, badblocks for storage, and stress tests for CPU. If the system passes all diagnostics but still fails, move to software. A useful trick: swap identical hardware between a failing and a working system. If the problem follows the hardware, you have your answer.
What’s the simplest logging strategy for a system with very limited storage?
Use a ring buffer in memory, as described above. Log only at ERROR or WARN level to persistent storage. For DEBUG and INFO, keep them in the ring buffer and flush only on error. If storage is extremely limited (e.g., an embedded device with 16MB flash), log to a remote syslog server over UDP. UDP is fire-and-forget, so it won’t block your application if the network is slow, but you may lose some messages during a network outage.
How can I convince management to invest time in debugging instead of just rebooting?
Track the cost of not debugging. Every reboot causes downtime. Every downtime has a business cost—lost transactions, idle staff, or delayed decisions. Present the cumulative downtime over a month in terms they understand: money or service impact. Then present the debugging effort as an investment with a clear return. If you can show that a week of systematic debugging will prevent 20 hours of downtime per month, the case makes itself.
What tools do you recommend for capturing intermittent network issues?
For continuous monitoring, smokeping is excellent—it graphs latency and packet loss over time, making intermittent drops visible. For deep-dive analysis, a packet capture tool like tcpdump with a ring buffer file set (-W and -C flags) can capture the last few hours of traffic without filling the disk. When an incident occurs, stop the capture and analyze the pcap file in Wireshark. Look for TCP retransmissions, duplicate ACKs, and connection resets.