Intermittent failures are the worst kind of production bug. They don’t break the system outright. They nibble at the edges, corrupt a few records, and vanish before you can open a log viewer. In environments with limited bandwidth, aging hardware, and unreliable power—the kind of environments I work in across Africa, South Asia, and Latin America—these ghosts in the machine are not just an annoyance. They can quietly erode trust in a system that took years to build. I’m Felix Okonkwo, and this is the approach I’ve learned to take when a system starts failing only sometimes.
What We Mean by “Intermittent” in a Constrained Environment
An intermittent failure is a fault that occurs unpredictably and often cannot be reproduced on demand. In a well-resourced data center, you might blame a cosmic ray flipping a memory bit. In the environments I work with—rural health clinics, agricultural logistics platforms, mobile money agents in peri-urban areas—the causes are usually more mundane but harder to isolate. We are dealing with voltage sags that brown out a server but not the UPS monitoring it. We are dealing with 2G edge connections that drop mid-TCP-handshake. We are dealing with SD cards in single-board computers that degrade after 10,000 write cycles because the database is logging too aggressively.
Intermittent failures are not just a technical problem. They are a trust problem. If a health worker cannot rely on a patient record system to save data, they will revert to paper. If a farmer cannot reliably check market prices, they will stop using the app. The cost of an intermittent failure is not just the corrupted transaction; it is the user you lose forever.
Start with the Physical Layer: Power and Cabling
Before you grep a single log file, check the power. I have spent days chasing a bug that turned out to be a failing 12V adapter that dipped below 11.5V under load. The server’s voltage regulator handled it most of the time, but when the disk spun up for a write operation, the voltage sagged further and the SD card threw a silent I/O error. The application logged a timeout. The real culprit was a power supply that cost $4 to replace.
In many of the sites I support, the “server room” is a shelf in a back office. Power comes from a solar-charged battery bank with an inverter that produces a modified sine wave. Some switch-mode power supplies do not handle that waveform well. I have seen systems reboot randomly because the inverter’s waveform confused the power supply’s under-voltage protection circuit. A pure sine wave inverter or a DC-DC power supply designed for automotive use can eliminate a whole class of intermittent failures.
Check your cables too. Ethernet cables crimped without proper strain relief can cause intermittent link flaps. A loose SATA connector inside a ruggedized case can cause I/O errors that look like disk corruption. These are not glamorous problems, but they are common. In a constrained environment, the physical layer is often the weakest link.
Power Monitoring on a Budget
You do not need expensive PDUs with per-outlet monitoring. A simple USB voltage logger—or even a multimeter with data logging—can capture sags over time. If you are using a Raspberry Pi or similar single-board computer, you can monitor the internal voltage rail via a script that reads the PMIC registers. I have a cron job that logs the core voltage every minute. When a failure occurs, I correlate the timestamp with the voltage log. It has saved me weeks of guesswork.
Logging That Survives the Failure
If your application logs to the same disk that fails intermittently, you are logging into a black hole. I learned this the hard way with a PostgreSQL database that would freeze for 30 seconds under heavy write load. The logs showed nothing because the logger was also blocked waiting for the disk. The fix was to ship logs off-host in real time using a lightweight forwarder like syslog-ng or even a simple UDP socket. UDP is lossy, but it won’t block your application. For critical errors, I use a small ring buffer in memory that gets flushed to a remote collector when connectivity allows.
Structured logging is not a luxury. When you are sifting through logs over a 64 kbps satellite link, you need to filter by request ID, user ID, or transaction ID. I add a unique correlation ID to every incoming request at the edge of the system and pass it through every service call. That way, when a user reports “my transfer failed last Tuesday,” I can pull every log line related to that transaction, even if the failure was in a downstream service.
What to Log When Nothing Is Wrong
Intermittent failures often leave no trace because the system appears healthy most of the time. You need to log the absence of expected events. If a cron job is supposed to run every 5 minutes, log a warning when it doesn’t run. If a heartbeat message from a remote sensor is missing for 60 seconds, log that gap. These “negative events” are often the only clue that something failed silently.
Network Intermittency: The Hardest Nut to Crack
In many parts of Africa and South Asia, network connectivity is the primary source of intermittent failures. Mobile networks drop packets during handovers between towers. VSAT links fade during heavy rain. Fiber backhaul gets cut by road construction. Your application must be designed to survive these events, but debugging them requires a different approach.
I use a tool called mtr (My TraceRoute) to continuously monitor path quality between two endpoints. It combines traceroute and ping, showing packet loss and latency per hop over time. Running mtr in report mode for 24 hours can reveal patterns: packet loss that spikes every afternoon when the temperature rises, or latency that increases during business hours when the backhaul is congested.
For application-level visibility, I instrument every outbound HTTP call with a histogram of response times and error codes. When an intermittent failure occurs, I can check whether the downstream service was slow, returning errors, or completely unreachable. This is not complex; a simple array of counters in memory, exported to a metrics endpoint, is enough to diagnose most problems.
State Corruption: The Silent Killer
Some of the nastiest intermittent failures come from corrupted internal state. A counter overflows and wraps to zero. A cached value expires but the new value fails to load, leaving a null that the code does not handle. A database connection is returned to the pool with an uncommitted transaction, and the next user of that connection sees stale data.
These bugs are intermittent because they depend on the exact sequence of operations. They survive testing because unit tests mock the database and integration tests run with a fresh state. In production, the system runs for weeks, accumulating edge cases. I have found that adding assertions to the code—even in production—is the most effective way to catch these. Assert that a counter is non-negative. Assert that a cached value is not null before using it. When an assertion fails, log the full context and reset the state. This is not elegant, but it prevents silent corruption.
Database Connection Pooling Pitfalls
Connection pools are a common source of intermittent failures in resource-constrained environments. The default pool sizes in many frameworks are tuned for a well-connected data center, not a satellite link with 600ms latency. When the pool is exhausted, requests queue up and eventually time out. The application logs show “connection timeout,” but the root cause is a slow upstream query that held connections too long.
I set connection pool sizes based on Little’s Law: L = λ × W, where L is the number of connections needed, λ is the request arrival rate, and W is the average time a connection is held. On a high-latency link, W is large, so you need more connections to maintain throughput. But more connections increase memory pressure. It is a trade-off. I also set aggressive statement timeouts and idle-in-transaction timeouts to prevent connections from being held indefinitely.
Reproducing the Unreproducible
You cannot fix what you cannot reproduce. But in a constrained environment, you often cannot reproduce the exact conditions that caused the failure. My approach is to build a “chaos lite” test setup that simulates the common failure modes: network latency, packet loss, disk I/O delays, and process kills. I do not need a full chaos engineering platform. A few shell scripts that use tc (traffic control) to add latency and packet loss, and stress-ng to consume CPU and memory, are enough to expose most race conditions and timeout bugs.
I also record production traffic when possible. A simple tcpdump filter that captures traffic to a specific port, rotated hourly, can be replayed against a staging environment. This is especially useful for debugging intermittent protocol errors. I once found a bug where a mobile network operator’s HTTP proxy was injecting duplicate Content-Length headers, which caused our HTTP client to fail parsing on about 1% of requests. Without a packet capture, I would never have found it.
Observability Without the Overhead
In resource-constrained environments, you cannot run a full Elasticsearch-Logstash-Kibana stack on-site. The hardware cannot handle it, and the bandwidth to ship logs to the cloud is too expensive or unreliable. I use a combination of lightweight tools: Vector for log aggregation (it uses a fraction of the resources of Logstash), Prometheus for metrics (single binary, efficient storage), and Grafana for dashboards that run on a Raspberry Pi. For tracing, I use Jaeger with the badger storage backend, which does not require an external database.
The key is to instrument the right things. Do not collect every metric just because you can. Focus on the “golden signals” for each service: request rate, error rate, and latency. Add business-level metrics that matter to your users: number of transactions completed, number of records synced, number of patients registered. When an intermittent failure occurs, these business metrics will show a dip even if all technical metrics look normal.
Case Study: The Disappearing SMS
A maternal health platform I worked on in Nigeria used SMS reminders for prenatal appointments. Intermittently, some reminders were not delivered. The SMS gateway reported successful delivery, but recipients never received them. The failure rate was about 2%, and it took us three weeks to find the root cause.
We instrumented every step: message queued, message sent to gateway, gateway acknowledgment received, delivery report received. We logged the mobile network operator (MNO) for each message. The pattern emerged: failures clustered on one MNO during peak hours. That MNO was silently dropping messages when its SMSC was overloaded, but still returning positive delivery acknowledgments. The fix was to switch to a different SMS route for that MNO during peak hours. Without the per-MNO logging, we would never have found the pattern.
Designing for Debuggability
The best time to prepare for intermittent failures is before they happen. I design systems with “debug endpoints” that expose internal state: current queue depths, connection pool status, cache hit rates, and the last N errors. These endpoints are protected by authentication and not advertised, but they are invaluable when troubleshooting. I also include a “health” endpoint that returns not just “OK” but a detailed breakdown of each dependency’s status.
In one deployment, we had a health check that verified database connectivity by running a SELECT 1. The check always passed, but the application was failing because a specific table was locked. We changed the health check to run a representative query against each critical table. Now we catch lock contention before users notice.
When to Escalate and When to Let Go
Not every intermittent failure is worth fixing. In a resource-constrained environment, you must triage. I use a simple framework: if the failure affects revenue, patient safety, or data integrity, it gets top priority. If it is cosmetic or affects only internal users, it goes into the backlog. If the cost of fixing it exceeds the cost of the failure over the system’s expected lifetime, I document it and move on.
This last point is controversial. Engineers want to fix every bug. But when you have one server, a part-time sysadmin, and a 128 kbps uplink, you must be ruthless about where you spend your limited debugging hours. Document the failure mode, add monitoring to detect it, and build a manual workaround. Sometimes, the most pragmatic solution is a cron job that restarts the service every night.
FAQ
Why do intermittent failures seem to happen more often in resource-constrained environments?
Resource-constrained environments amplify small instabilities. A voltage fluctuation that would be absorbed by a high-end power supply can crash a low-cost single-board computer. A network hiccup that would be retried transparently on a fiber connection can cause a TCP timeout on a high-latency satellite link. The margins are thinner, so failures that would be rare in a well-resourced data center become common.
What is the single most effective tool for debugging intermittent failures?
There is no single tool, but if I had to choose one, it would be structured logging with correlation IDs. Without the ability to trace a single transaction across services and time, you are debugging in the dark. Correlation IDs let you connect a user complaint to the exact set of log lines that describe what happened, even if the failure occurred hours or days ago.
How do I convince stakeholders to invest time in debugging intermittent failures?
Frame the cost in terms they understand. Calculate the number of failed transactions per month and multiply by the value of each transaction. For a health system, estimate the number of missed appointments and the health outcomes that result. For a financial system, calculate the direct revenue loss plus the cost of manual reconciliation. Intermittent failures have a measurable business impact; your job is to make that impact visible.
Can intermittent failures be prevented entirely?
No. In any complex system, some failures will be intermittent. The goal is not perfection but resilience. Design systems that degrade gracefully, detect failures quickly, and recover automatically. Invest in monitoring that alerts you before users notice. And accept that some failures will remain mysterious. Document them, learn from them, and move on.
Next Steps for Your System
If you are dealing with intermittent failures right now, start with the physical layer. Check your power, your cables, your storage media. Then add correlation IDs to your logging. Then instrument your outbound network calls. These three steps will catch the majority of intermittent failures in constrained environments. The rest require patience, a methodical approach, and a willingness to accept that not every ghost can be exorcised.
This article is part of a series on operating production systems in non-ideal environments. Future pieces will cover backup strategies for intermittent connectivity, monitoring on a shoestring budget, and designing self-healing systems that can survive without constant human attention.