Intermittent failures are the most expensive kind of production problem. They are not the clean crash that pages you at 2 a.m. and points to a stack trace. They are the request that fails once in every 400 calls, the batch job that dies on the third Tuesday of the month, the mobile money transaction that times out only when the network is congested and the database is under load. In systems engineering for resource-constrained environments — the kind I work with across Nigeria, Ghana, Kenya, and parts of South Asia — intermittent failures are also the most common. Power dips, shared infrastructure, oversubscribed links, and hardware that is older than the intern who wrote the deployment script all combine to create failures that refuse to reproduce on demand.
This article is about how to debug those failures without pretending you have a lab environment, a dedicated observability team, or unlimited time. I will cover the mental model, the data you need to collect, the tools that work when bandwidth and disk are tight, and the trade-offs you accept when you cannot instrument everything. The goal is not to eliminate intermittent failures — that is a fantasy in non-ideal infrastructure. The goal is to make them explainable, and then to make them rare enough that your team can sleep.
What an Intermittent Failure Actually Is
An intermittent failure is a failure that occurs under conditions you have not yet identified. That is the honest definition. The word “intermittent” is a label for your ignorance, not a property of the system. Once you know the conditions, the failure becomes deterministic: it happens when the disk queue depth exceeds 40, when the upstream API returns a 502 after 3 seconds, when the voltage drops below 200V and the UPS switches to battery. The debugging job is to convert “sometimes” into “when.”
In resource-constrained environments, the conditions are often environmental before they are logical. A server in Lagos does not fail the same way a server in Frankfurt fails. Heat, dust, generator transfer switches, ISP peering disputes, and SIM card registration expiries all sit inside the failure chain. If you start by assuming the failure is in your code, you will waste days. If you start by mapping the physical and network path, you will often find the trigger faster.
Build a Timeline Before You Build a Theory
The first mistake engineers make with intermittent failures is to jump to a hypothesis. “It must be the connection pool.” “It must be memory pressure.” “It must be the load balancer.” Maybe. But a hypothesis without a timeline is a guess. You need to know exactly when the failure happened, what else was happening at that moment, and what changed in the minutes before.
In a well-instrumented system, you pull logs and metrics and correlate timestamps. In a resource-constrained system, you may not have centralized logging. You may have logs rotating every 24 hours on the server itself, or no logs at all because the disk is full. So you build the timeline from whatever you have: application logs, web server access logs, database slow query logs, cron job output, SMS gateway delivery reports, even the timestamps on support tickets from users. The timeline is the skeleton. Everything else hangs on it.
One practical technique: when a user reports an intermittent failure, ask for the exact time, the phone number or account ID, the amount or action, and the network they were on. Do not ask “what did you see?” — users will tell you a story. Ask for the timestamp. Then go to the logs and find the request. If the request is not in the logs, that is itself a finding: the failure happened before your application saw the request, or your logging is incomplete.
Instrument the Boundaries, Not Just the Code
Most intermittent failures in production happen at boundaries: between your application and the database, between your application and an external API, between the mobile network and your server, between the power grid and your UPS. If you only instrument inside your code, you will see the symptom but not the cause. You need to instrument the edges.
At a minimum, log the following for every external call:
- The target host and port
- The start time and end time, in milliseconds
- The HTTP status code or error code
- The number of retries, if any
- The size of the request and response payloads
This is not expensive. A single structured log line per external call costs a few hundred bytes. If you are running on a VPS with a 40GB disk, you can store months of these logs. The value is enormous: when the failure happens, you can see whether the external call was slow, failed, or never returned. You can see whether the failure clustered around a particular upstream provider or a particular time of day.
For database calls, log the query duration and the number of rows returned. For file system operations, log the path and the duration. For network operations, log the source and destination IP and the round-trip time. The pattern is the same: record the boundary crossing, not just the outcome.
Use Sampling When You Cannot Store Everything
In resource-constrained environments, you cannot log every request at debug level. Disk fills up, I/O slows down, and the logging itself becomes a source of intermittent failure. The answer is sampling. Log 100% of errors, 10% of slow requests, and 1% of normal requests. Or log every Nth request deterministically, so you can reconstruct a representative sample without storing the world.
Sampling has a trade-off: you may miss the exact request that failed. But if you sample consistently, you will still see the pattern. If 1% of requests are sampled and the failure rate is 0.25%, you will see roughly one failed request for every 400 sampled requests. That is enough to correlate with other signals. The alternative — logging everything until the disk fills and the server crashes — is worse.
One trick I use: when a request fails, write a full trace for that request, including the previous 50 requests in the same session or from the same IP. This gives you context without storing full traces for every request. It is a poor man’s distributed tracing, and it works surprisingly well.
Correlate with Infrastructure Signals
Intermittent failures in Lagos or Nairobi or Dhaka often correlate with infrastructure signals that have nothing to do with your code. Power quality is the big one. A voltage sag can cause a server to reboot, a disk to corrupt a write, or a network switch to reset. If you are not monitoring power, you are debugging blind.
Cheap ways to monitor power: a UPS with a USB or network interface that logs transfer events; a smart plug that reports voltage and frequency; a Raspberry Pi with a voltage sensor. You do not need a data center-grade power monitor. You need a timestamped record of when the power did something unusual. Then you correlate that with your application logs.
Network quality is the second signal. Use a tool like SmokePing or a simple cron job that pings your upstream providers and logs latency and packet loss. When your application fails intermittently, check whether the network was also misbehaving at that moment. In many African and South Asian markets, international traffic routes through a small number of undersea cables and exchange points. A cable cut or a peering dispute can cause intermittent failures for hours, and your application logs will show timeouts to external APIs with no other explanation.
Disk health is the third signal. A failing disk produces intermittent read and write errors long before it dies completely. Use SMART monitoring if your disks support it. If you are on cloud infrastructure, watch the disk queue depth and I/O wait metrics. A disk that is 90% full will also cause intermittent failures when the filesystem has to work hard to find free blocks.
Reproduce the Failure by Shrinking the System
You cannot always reproduce an intermittent failure in production, but you can often reproduce it in a smaller version of the system. The key is to shrink the system without changing the conditions that matter. If the failure happens under load, generate load. If it happens when the network is slow, add artificial latency. If it happens when the disk is full, fill the disk to 90% and try again.
This is where many engineers give up. They say, “I cannot reproduce it in my development environment, so I cannot fix it.” That is the wrong frame. Your development environment is not the production environment. The question is not whether you can reproduce it on your laptop. The question is whether you can reproduce it in a test environment that shares the relevant constraints: the same database version, the same network latency profile, the same memory limits, the same disk type.
In resource-constrained environments, you may not have a separate test environment. You may have to test in production, carefully. That is not ideal, but it is honest. If you must test in production, do it during low-traffic hours, with a canary deployment or a feature flag, and with a rollback plan. The alternative — shipping a fix based on a guess and hoping — is worse.
Common Causes of Intermittent Failures in Non-Ideal Infrastructure
Over the years, I have seen a small set of causes account for most intermittent failures in the environments I work in. They are not exotic. They are boring, and that is the point.
Connection Pool Exhaustion
Your application opens a connection to the database, the connection times out or is dropped by a firewall, and the pool does not reclaim it. Over time, the pool fills with dead connections. New requests wait for a connection, time out, and fail. The failure is intermittent because it only happens when the pool is exhausted, which depends on traffic patterns and how often connections are dropped.
The fix is not to make the pool bigger. The fix is to set a connection timeout, a validation query, and a maximum lifetime for connections. In a flaky network, set the maximum lifetime to something short — 15 or 30 minutes — so dead connections are recycled before they accumulate.
DNS Resolution Failures
Your application calls an external API by hostname. The DNS resolver times out or returns a stale record. The call fails. The next call succeeds because the resolver cached a good record. This is maddening to debug because the failure looks random. The fix is to log DNS resolution time separately from connection time, and to use a local caching resolver like dnsmasq or systemd-resolved with a short negative cache TTL.
Time Synchronization Drift
Your servers’ clocks drift apart. A request is timestamped at 10:00:01 on one server and 09:59:58 on another. When you correlate logs, the timeline does not line up. Worse, if you use time-based tokens or signatures, a clock that is off by a few seconds can cause intermittent authentication failures. Run NTP everywhere, and monitor the offset. In environments with unreliable internet, use multiple NTP servers and a local time source if possible.
File Descriptor Limits
Your application opens files or sockets and does not close them. Eventually it hits the file descriptor limit and starts failing. The failure is intermittent because it only happens after the process has been running for a while and has accumulated enough leaked descriptors. The fix is to monitor file descriptor usage and to set the limit high enough that you have headroom, but not so high that you mask the leak.
Memory Pressure and the OOM Killer
Your server runs out of memory, the kernel’s OOM killer terminates a process, and the process restarts. Requests that were in flight fail. The failure is intermittent because it only happens when memory pressure peaks, which depends on traffic and on what else is running on the box. The fix is to monitor memory usage and to set appropriate memory limits for each process, so the OOM killer terminates the right process instead of a random one.
Write the Postmortem Before You Find the Cause
This sounds backwards, but it works. When an intermittent failure appears, write the postmortem as if you already know the cause. Write the timeline, the impact, the actions taken, and the open questions. The act of writing forces you to identify what you do not know. Those gaps become your debugging plan.
For example, you might write: “At 14:32 UTC, 12 requests to the payment API failed with timeout errors. The application logs show the requests were sent but no response was received. The database was not under load. The network monitoring shows a 2-minute period of elevated latency to the upstream provider starting at 14:31. Open question: why did the upstream provider’s latency spike?” That open question is specific. You can investigate it. You can ask the provider. You can check their status page. You can look at historical patterns.
If you skip the postmortem and just poke at logs, you will wander. The postmortem is a debugging tool, not a bureaucratic ritual.
Tools That Work When Bandwidth and Disk Are Tight
You do not need a commercial observability platform to debug intermittent failures. You need a few tools that are cheap, reliable, and easy to run on modest hardware.
- Journald and rsyslog for local log collection. They are already on most Linux systems. Configure them to rotate logs aggressively and to forward critical errors to a central server if you have one.
- Prometheus and Grafana for metrics. They are free, they run on a small VPS, and they can scrape metrics from your applications and from node exporters. If you have never used them, start with a single node exporter and a single dashboard.
- SmokePing for network latency monitoring. It is old, it is ugly, and it works. It will show you exactly when the network got slow.
- tcpdump for packet capture. When everything else fails, capture packets on the server and look at what actually went over the wire. In a resource-constrained environment, capture only the traffic to the failing service, and rotate the capture files aggressively.
- strace and ltrace for system call tracing. They are heavy, so use them sparingly, but they can reveal the exact system call that is failing when your application gives you nothing useful.
The common thread: these tools are boring, they are well-documented, and they do not require a SaaS subscription. In an environment where the power can go out at any moment, boring tools are a feature.
Accept the Trade-Offs
You cannot debug intermittent failures the way a well-funded team in a stable data center does. You do not have unlimited log retention. You do not have a staging environment that mirrors production. You do not have a vendor on call. You have to make trade-offs.
The biggest trade-off is between logging volume and disk space. You cannot log everything. You have to choose what to log, and you have to accept that you will miss some failures. The second trade-off is between investigation time and user impact. You cannot keep a failing system running while you debug it forever. At some point, you have to restart the service, clear the queue, or fail over to a backup, even if that destroys the evidence. The third trade-off is between fixing the root cause and applying a workaround. In a resource-constrained environment, a workaround that keeps the system running is often the right call, as long as you document it and schedule the root-cause fix.
None of these trade-offs are comfortable. But pretending they do not exist is worse. The honest engineer says: “I do not know why this failed, but I know when it failed, I know what else was happening, and I have a plan to find out. In the meantime, here is a workaround that keeps the system alive.”
Build a Runbook for the Next Intermittent Failure
The best time to prepare for an intermittent failure is before it happens. Write a runbook that your team can follow when the next one appears. The runbook should include:
- How to collect the application logs for the affected time window
- How to check the database slow query log
- How to check the network latency monitor
- How to check the power monitor
- How to check disk health and file descriptor usage
- How to take a packet capture without filling the disk
- How to write the postmortem
This runbook is not a substitute for thinking. It is a checklist that prevents you from forgetting the basics when you are under pressure. In a resource-constrained environment, the basics are often enough to find the cause.
Frequently Asked Questions
Why do intermittent failures happen more often in resource-constrained environments?
Because the infrastructure itself is less stable. Power quality varies, network links are oversubscribed, hardware is older, and redundancy is limited. A small voltage sag or a brief network congestion event can cause a failure that would never happen in a stable data center. The application code may be perfectly fine; the environment is the trigger.
How do I debug an intermittent failure when I have no logs?
Start by adding logs. You cannot debug what you cannot see. Add structured logging at the boundaries: external API calls, database queries, file system operations. Log the timestamp, the duration, the status, and the target. Then wait for the failure to happen again. In the meantime, use whatever indirect evidence you have: user reports with timestamps, network monitoring, power monitoring, and server metrics.
What is the most common cause of intermittent failures in production systems?
In my experience, the most common cause is a boundary failure: a connection to a database or an external API that times out, is dropped, or is exhausted. Connection pool exhaustion, DNS resolution failures, and network latency spikes account for a large share of intermittent failures. The second most common cause is resource exhaustion: memory pressure, file descriptor leaks, or disk full conditions.
Should I use a distributed tracing system to debug intermittent failures?
If you have the resources to run one, yes. Distributed tracing gives you a request-level view that logs alone cannot provide. But in resource-constrained environments, a full tracing system may be too heavy. Start with structured logging at the boundaries and a correlation ID that ties related requests together. That gives you 80% of the value at 20% of the cost.
Next Steps for This Blog
This article is the first in a series on production debugging in non-ideal infrastructure. The next article will cover how to set up lightweight monitoring with Prometheus and Grafana on a small VPS, including what to monitor when you have limited disk and bandwidth. If you have a specific intermittent failure you are fighting, send me the details — the timeline, the symptoms, and what you have tried — and I will use it as a case study in a future post.