Intermittent failures are the ghost in the machine. They show up under load, disappear the moment you attach a debugger, and resurface at 3 a.m. when the only person on call is you. In places where bandwidth is measured in kilobits, power is a negotiation, and hardware is whatever you could find at the market last Tuesday, these ghosts aren’t just annoying—they can shut down a clinic’s patient records system or silence mobile money transfers for an entire village. I’m Felix Okonkwo, and I’ve spent the better part of a decade chasing these phantoms across West African server rooms and East African cloud deployments. This article is about the practical, sometimes messy, methods that actually work when you can’t just “spin up a new instance” or “check the logs in Splunk.”
Why Intermittent Failures Hit Different in Constrained Environments
In a well-funded data center, debugging an intermittent failure often means throwing resources at the problem: replicate the production traffic on a staging cluster, attach a low-overhead profiler, or comb through terabytes of structured logs. In the environments I work in—rural health clinics, microfinance offices, remote agricultural processing hubs—none of that is possible. The production server is also the staging server. Logs rotate every few hours because disk space is tight. The internet connection is a 3G modem that drops when it rains. The failure you’re chasing might be a software bug, but it’s just as likely to be a corroded RAM slot, a voltage sag from a generator switchover, or a DNS timeout because the ISP’s resolver is overloaded.
This means your mental model has to stretch. You’re not just debugging code; you’re debugging a socio-technical system. The intermittent failure is a signal from that system, and your job is to interpret it with limited tools.
Start with a Hypothesis, Not a Tool
The most common mistake I see is reaching for a tool before forming a clear hypothesis. You think, “I’ll install New Relic,” or “I’ll add more logging.” But in a constrained environment, every additional agent consumes precious CPU and memory. Every extra log line fills the disk faster. Before you change anything, write down what you think is happening. Be specific: “I suspect the payment processing service fails when the database connection pool is exhausted during the 11 a.m. bulk settlement run.” A good hypothesis is falsifiable and narrow.
Then ask: what’s the cheapest, least invasive way to test this? Often, it’s not a new tool but a clever use of existing ones.
Use What the OS Already Gives You
Linux and Windows both ship with powerful introspection tools that are often overlooked. They’re lightweight, well-documented, and already installed.
Using sar for Historical System Metrics
The sysstat package, which includes sar, is a lifeline. It collects CPU, memory, I/O, and network stats at regular intervals and stores them in binary logs. When a user reports that “the system was slow yesterday around 2 PM,” you can query sar to see exactly what was happening. Look for spikes in disk I/O wait times, sudden drops in available memory, or unusual network retransmission rates. On one deployment in northern Nigeria, we traced a daily 10-minute outage to a cron job that ran updatedb at noon, thrashing the single disk. The fix was a one-line change to the crontab.
Tracking Down Resource Leaks with pidstat
Intermittent slowdowns often come from a process that gradually leaks memory or file handles. pidstat can track a specific process over time and log the data. I once debugged a Python service that would hang every three days. pidstat showed file descriptors climbing steadily. The culprit was a library that opened a new HTTP connection for each request but never closed them when the remote end reset. The fix was a two-line patch, but finding it without pidstat would have meant waiting for the crash and guessing.
Network Blind Spots: ss and tcpdump
Intermittent network failures are common in environments with unreliable last-mile connectivity. Before blaming the ISP, check your own house. ss -s gives a quick summary of socket states. A high number of TIME_WAIT sockets can indicate connection churn. tcpdump with a rotating capture file (using -C and -W) can run for days with minimal overhead. I once captured a pattern where a remote API would send a TCP RST exactly 30 seconds after a request if the backend was overloaded. The application interpreted this as a network failure and retried, making the overload worse. The capture file was 2 MB and held the answer.
Designing for Debuggability When You Can’t Afford Observability
Full observability stacks—think Prometheus, Grafana, ELK—are wonderful. They also need RAM, disk, and network bandwidth that may not exist. In their absence, you have to bake debuggability into the application itself. This isn’t about adding thousands of log lines; it’s about strategic, structured signals.
Structured Logs with a Fixed Schema
Plain text logs are easy to write but hard to parse when you’re in a hurry. Even if you can’t ship logs to a central server, write them as JSON. Include a timestamp, a severity level, a correlation ID, and a message. When a failure occurs, you can use grep and jq to filter and analyze the log file directly on the server. A 50 MB JSON log file is searchable with grep; a 50 MB unstructured text file is a haystack.
Correlation IDs Across Boundaries
An intermittent failure often involves multiple services. A request comes in, hits an API gateway, calls an authentication service, then a database. If any step fails, you need to trace it back. Generate a unique correlation ID at the entry point and pass it through HTTP headers or message metadata. Log it at every step. When a user reports an error, ask them for the approximate time and any error code, then grep for that correlation ID. This is a poor person’s distributed tracing, and it works.
Health Check Endpoints That Actually Check Health
A health check that returns “200 OK” because the process is alive is useless. Your health check should verify the things that actually fail: database connectivity, disk space, memory allocation, dependent service reachability. But be careful—a health check that runs a heavy query can itself cause an outage. Keep it cheap: ping the database, check that a critical file is readable, verify that the system clock is sane. Expose this as a simple JSON endpoint. When you suspect a problem, you can hit it from a remote monitoring script or even a browser on a phone.
Reproducing the Unreproducible
Intermittent failures are hard because they resist reproduction. But “intermittent” doesn’t mean “random.” It means the trigger is hidden. Your job is to find the trigger.
Traffic Shadowing with Limited Resources
You can’t duplicate production traffic to a staging environment if you have no staging environment. But you can replay a sample. Tools like tcpreplay or GoReplay can capture a slice of production traffic and replay it against a test instance on a different port on the same machine. This is risky—do it during low-traffic periods and monitor resource usage. The goal isn’t to replicate the full load but to find the specific request pattern that triggers the bug.
Chaos Engineering on a Shoestring
Chaos engineering sounds like a luxury for Netflix. But the core idea—deliberately injecting failure to test system resilience—can be done with shell scripts. Write a script that randomly kills a process, drops a network connection, or fills a disk partition. Run it in a controlled way during a maintenance window. The goal isn’t to break production but to verify that your monitoring catches the failure and your recovery procedures work. I’ve found more bugs by simulating a full disk than by any code review.
When the Hardware Is the Suspect
In resource-constrained environments, hardware is often reused, refurbished, or exposed to harsh conditions—dust, heat, unstable power. Intermittent failures that defy software explanation often have a physical root cause.
Power Supply Instability
Voltage sags and spikes can cause CPU errors, disk corruption, and random reboots. If you’re not using a line-interactive UPS or an inverter with a stable sine wave output, your “software” bug might be a power quality problem. A simple mains power monitor that logs voltage over time can reveal patterns. I once traced a server’s weekly crash to the exact time a nearby factory switched its heavy machinery on and off.
Thermal Throttling and Dust
In hot, dusty environments, CPU throttling is common. When the processor slows down to prevent overheating, timeouts cascade. Applications that work perfectly in the morning fail in the afternoon heat. Check your system logs for CPU frequency scaling messages. Clean the fans and heatsinks. If the server is in a closed room without ventilation, a simple exhaust fan can be more effective than a software patch.
Building a Lightweight Debugging Toolkit
Over the years, I’ve assembled a small set of scripts and tools that I carry on a USB stick or keep in a private Git repository. These aren’t complex programs; they’re wrappers around standard Unix tools that save time when you’re on site and the pressure is on.
- log-grep.sh: A script that searches across multiple log files, filters by time range, and highlights patterns like “error,” “timeout,” or “refused.” It also counts occurrences to spot spikes.
- quick-profile.sh: Uses
perforstraceto attach to a running process for 60 seconds and output the top syscalls or kernel functions. Useful when a process suddenly goes CPU-bound. - conn-watch.sh: Polls
ssandnetstatevery few seconds and logs changes in connection states. Helps catch socket leaks or port exhaustion as they happen. - disk-health.sh: Checks SMART data, inode usage, and disk space, then sends an alert if any threshold is crossed. Many intermittent failures start with a disk that is quietly failing.
Communication During an Outage
Debugging isn’t just a technical process; it’s a social one. When a system is down, stakeholders want to know what’s happening. In constrained environments, you may not have Slack or a status page. What you do have is WhatsApp, SMS, or a physical whiteboard in the office. Establish a single point of truth. Update it on a schedule, even if the update is “still investigating.” This reduces the flood of “is it fixed yet?” messages and lets you focus.
Be honest about what you know and what you don’t. If the problem is a generator that ran out of diesel, say so. If you don’t know the cause, say that too, but give a time when you’ll provide the next update. This builds trust and buys you the space to work.
Postmortems That Actually Prevent Recurrence
A postmortem isn’t a document to satisfy a manager. It’s a tool for your future self, who will face a similar failure at 2 a.m. six months from now. Write it so that a tired, stressed version of you can follow it. Include the exact commands you ran, the log excerpts that confirmed the hypothesis, and the fix you applied. Store it in a place that’s accessible even when the main system is down—a printed notebook, an offline wiki, a text file on a phone.
I keep a “Blackout Book” in every server room I manage. It contains network diagrams, IP addresses, console cables, and printed postmortems of past outages. When the lights go out and the UPS is beeping, that book is worth more than any monitoring dashboard.
FAQ
What’s the first thing I should check when a production system starts failing intermittently?
Check the system resources: CPU load, memory usage, disk I/O, and network sockets. Use tools like top, free, iostat, and ss. Look for any resource that’s saturated or close to its limit. In constrained environments, resource exhaustion is the most common trigger for intermittent failures. Also, check the system clock—time drift can cause authentication failures and data corruption.
How can I debug a problem that only happens once a week without setting up complex monitoring?
Use sar to collect system metrics continuously. It uses negligible resources and keeps days of history. When the failure occurs, you can look back at the exact time and see what changed. Also, enable persistent logging for your application with rotation. A simple cron job that archives logs to a compressed file can preserve weeks of data on a small disk. Finally, ask users to note the exact time of the failure—this is often the most reliable trigger for your investigation.
What if the failure is caused by the ISP or mobile network, and I have no control over it?
Design your application to be resilient to network failures. Implement retry logic with exponential backoff and jitter. Use local queues that can store requests when the network is down and forward them when it returns. For critical services, consider a multi-homed setup with two different ISPs, even if one is a low-bandwidth backup. Test your failover regularly. And keep a log of network outages—this data can help you negotiate service credits or justify an upgrade to management.
Next Steps for Your Own Systems
This article is part of a series on operating production systems in challenging environments. The next piece will cover backup strategies when cloud storage isn’t an option and your backup window is measured in hours, not minutes. If you have a specific failure scenario you’d like me to analyze, send a message through the contact page. I read every one, though my responses may be delayed by the same infrastructure constraints we’re all working to overcome.