There is a special kind of frustration that comes from a system that only fails when you are not looking. In the environments I work with—off-grid solar monitoring in rural Tanzania, last-mile logistics platforms in Bihar, community health worker apps in the Peruvian Amazon—the failures are rarely spectacular. They are quiet. A sensor stops reporting for three hours, then resumes. A payment gateway times out only on Tuesdays. A health worker’s sync fails when she moves between two specific cell towers. These are not bugs you can reproduce on a staging server in a climate-controlled data center. They are bugs born from the friction between software and a messy, constrained physical world.
This article is about debugging those intermittent failures in production, when you cannot simply spin up a clone of the environment, when your logging budget is a rounding error compared to your bandwidth bill, and when the root cause is often not a null pointer but a generator running out of diesel. I will walk through the mental model I use, the tools that actually work in these settings, and the trade-offs you make when perfect observability is a fantasy.
First, Define the Shape of the Failure
Before you touch a log file, you need to characterize the failure. I use a simple framework I call the “Three T’s”: Timing, Topology, and Trigger. This is not a formal methodology; it is a set of questions I have learned to ask after too many nights staring at dashboards that told me nothing.
Timing: When Does It Happen?
Intermittent failures often have a rhythm. In one deployment of an inventory management system for rural clinics in Uganda, we saw sync failures spike every Monday morning. The obvious suspect was server load—everyone returning from the weekend. The real cause was that the clinic’s solar batteries were deeply discharged after two days of no sun and heavy phone charging by staff. The server was on a timer that cut power at 8 AM to protect the batteries, right when the sync job ran. The timing pattern was weekly, but the trigger was environmental, not digital.
Ask: Is the failure periodic? Does it correlate with time of day, day of week, or month? In many off-grid sites, power availability follows a daily solar cycle, and generator refueling happens on a schedule. Network congestion often tracks with local market days or school hours. These patterns are invisible to APM tools but obvious if you talk to the site operator.
Topology: Where Does It Happen?
Map the failure to physical or logical locations. Is it one device, one cell tower, one region? In a project tracking vaccine deliveries in northern Nigeria, we saw intermittent GPS data loss. It was not random—it clustered around specific waypoints. The cause was not software; it was that those waypoints were under thick tree canopies that blocked the GPS signal. The fix was not code but a firmware update that increased the GPS timeout window.
In resource-constrained environments, topology often reveals infrastructure gaps. A failure that only occurs on devices connected to a particular mobile network operator might be caused by that operator’s aggressive NAT timeout, not your application logic. A failure that only happens on devices with less than 15% battery might be caused by the OS killing background services. These are not bugs you can fix with a better algorithm; they are constraints you must design around.
Trigger: What Changed?
Intermittent failures are often triggered by a state change. The challenge is that the trigger may be external and unmonitored. I once spent two weeks chasing a bug where a health worker’s tablet would fail to upload forms. The trigger was that she plugged in the device to charge while the app was open, causing a USB debugging prompt that blocked the upload thread. The fix was a single line in the Android manifest. The trigger was invisible in our server logs.
Triggers can be environmental (power fluctuation, temperature, humidity), operational (a new batch of SIM cards, a firmware update pushed by the manufacturer without notice), or behavioral (a user following a workflow you never tested). In constrained environments, you must assume that the trigger is something you are not currently measuring.
Building a Debugging Toolkit for Constrained Environments
When you cannot afford Datadog or New Relic, and your edge nodes run on 2G with frequent power cuts, you need a different approach. Here are the tools and techniques I have found most useful.
Structured Logging with a Purpose
Logs are your primary forensic tool, but in bandwidth-constrained environments, you cannot ship everything. I use a pattern I call “graduated logging”: by default, devices log only errors and critical state transitions. When a user reports an issue, we can remotely increase the log level for that specific device for a limited time window. This keeps baseline data usage low while allowing detailed debugging when needed.
Every log entry must include a correlation ID that ties together a single user session or transaction. In intermittent failure scenarios, this is non-negotiable. Without it, you are trying to assemble a jigsaw puzzle in the dark. I also recommend logging key environmental metrics: battery level, signal strength (RSSI), network type (2G/3G/4G), and available memory. These are often the real culprits.
Lightweight Tracing with OpenTelemetry
Distributed tracing is not just for microservices in Kubernetes. Even a simple mobile app talking to a cloud backend can benefit from tracing, especially when failures span network boundaries. We have used OpenTelemetry with a sampling rate of 1% to keep overhead low. The key is to propagate trace context through unreliable transports—SMS, USSD, or store-and-forward queues. This lets you see exactly where a request died, even if it was in a queue on a device that was off for three days.
In one deployment, tracing revealed that a “server timeout” was actually a 45-second delay in a GSM modem’s AT command response. The modem was overheating in a metal enclosure under the sun. No amount of backend optimization would have fixed that.
Heartbeats and Watchdogs
When you cannot afford continuous monitoring, use heartbeats. A simple periodic message from the device to the server saying “I am alive, here is my status” can detect silent failures. If a device misses two heartbeats, trigger an alert. The heartbeat payload should include the device’s local time, battery level, pending job count, and last error code. This is cheap to send (a few bytes over SMS or UDP) and gives you a rough health dashboard.
Watchdogs are the device-side equivalent. A hardware watchdog timer can reboot a hung device. A software watchdog can restart a stuck process. In one deployment of Linux-based gateways in rural clinics, we used a simple script that checked if the main application had updated a timestamp file in the last five minutes. If not, it killed and restarted the process. Crude, but it kept the system alive until we could push a proper fix.
Reproducing the Unreproducible
You cannot replicate a brownout in a London data center. But you can simulate the conditions that trigger intermittent failures. I keep a “chaos bench” in my workshop: a Raspberry Pi connected to a variable power supply, a network emulator that can throttle bandwidth and inject packet loss, and a heat gun. When a field device misbehaves, I try to recreate the environmental conditions reported by the user.
This is not sophisticated chaos engineering; it is pragmatic. I once debugged a payment terminal that would crash only when the ambient temperature exceeded 38°C. The heat caused the internal voltage regulator to droop, triggering a brownout reset. The fix was a firmware update that slowed the CPU clock at high temperatures. Without a heat gun and a thermocouple, I would still be staring at log files.
Observability on a Shoestring
Full-stack observability platforms are priced for Silicon Valley. When your annual IT budget is less than the cost of a single Datadog seat, you build your own. Here is a stack that has worked for me in multiple projects across East Africa and South Asia.
Metrics: Prometheus on a VPS
Prometheus is open source and runs happily on a $20/month VPS. Its pull model is ideal for environments where devices are not always reachable—you can configure Prometheus to scrape a push gateway that devices post to when they have connectivity. I use the textfile collector pattern: devices write metrics to a local file, and a simple script uploads that file to an HTTP endpoint when bandwidth is available. This decouples metric generation from transmission and survives intermittent connectivity.
Logs: Loki and the “Sneakernet”
Grafana Loki is a log aggregation system that indexes only metadata, making it far cheaper to run than Elasticsearch. In one deployment, we could not afford the bandwidth to ship logs from 200 remote gateways. Instead, we stored logs on USB drives and had field staff swap them during monthly maintenance visits. The drives were mailed to a central office, where logs were ingested into Loki. It was not real-time, but it was sufficient for post-mortem analysis of intermittent failures.
Alerting: When “Notify Me” Means an SMS
Email alerts are useless if the person who can fix the problem is in a field with no data coverage. I configure alerting to send SMS messages via a local aggregator like Africa’s Talking or Twilio’s SMS API. The alert must be terse: device ID, failure type, location. I also set up escalation policies that account for local time zones and working hours—waking someone at 3 AM for a non-critical failure burns goodwill fast.
The Human Debugger: Field Staff as Sensors
In the environments I work in, the most valuable debugging tool is a field officer with a notebook. They see things your telemetry cannot: a device placed too close to a metal roof that overheats, a user who charges the tablet from a faulty generator, a clinic where the staff share one power strip among five devices. I train field staff to record three things when a failure occurs: what they were doing, what the device did, and what the environment was like. This simple practice has solved more intermittent failures than any log analysis tool.
I also make it a point to visit sites myself, especially during the rainy season or just after a new deployment. There is no substitute for seeing the actual conditions. On one visit, I found that a “network outage” was caused by a goat chewing through an Ethernet cable. The logs just showed a link-down event. The goat did not leave a stack trace.
Designing for Debuggability
When you know that failures will be hard to reproduce, you must design the system to be debuggable from the start. This means building in “explainability” features that are often cut from commercial products because they add complexity. In constrained environments, they are not optional.
State Dumps and Flight Recorders
Every device should be able to produce a snapshot of its current state on demand: memory usage, active threads, pending queues, last successful sync timestamp, battery voltage, signal strength. I implement this as a simple HTTP endpoint on the device’s local network or a USSD code that returns the data via SMS. When a user reports a problem, support staff can request a state dump and attach it to the ticket. This is often enough to identify the issue without a site visit.
For harder problems, I use a circular buffer that continuously records key events—a poor man’s flight recorder. When a crash is detected, the buffer is persisted to disk. On next connectivity, it is uploaded to the server. This captures the moments leading up to a failure without the bandwidth cost of full logging.
Graceful Degradation and Partial Operation
In resource-constrained environments, a system that fails completely is worse than one that degrades gracefully. If the backend is unreachable, the device should continue to operate offline, queuing transactions locally. If GPS is unavailable, it should fall back to cell tower triangulation or manual location entry. If the battery is low, it should disable non-critical features. Each degradation should be logged and reported, so you can see patterns over time.
This is not just good UX; it is a debugging aid. When you see a spike in degraded-mode operations, you know something has changed in the environment—a cell tower is down, a solar panel is failing, or users are working longer hours. The degradation metrics become your leading indicators.
Case Study: The Vanishing Vaccine Data
Let me walk through a real example. We deployed a cold-chain monitoring system for vaccine refrigerators in rural health posts. The system used IoT sensors that reported temperature data via GPRS to a cloud dashboard. Intermittently, data from certain health posts would stop for 2-4 hours, then resume. No errors were logged on the server; the devices simply went silent.
We applied the Three T’s framework. Timing: the gaps occurred mostly between 2 PM and 6 PM. Topology: only health posts in one district were affected. Trigger: unknown. We sent a field officer to investigate. He found that the affected health posts were on a shared transformer that experienced voltage sags in the afternoon when the local maize mill operated. The IoT gateway’s power supply was sensitive to these sags and would reset, but the reset took 2-4 hours because the device’s firmware performed a full file system check on every boot.
The fix was twofold: replace the power supplies with wide-input-range models, and update the firmware to skip the file system check on warm boots. The intermittent failures disappeared. The logs never showed an error because the device lost power before it could write one. The solution came from understanding the local environment, not from analyzing code.
When to Stop Digging
Not every intermittent failure is worth solving. In resource-constrained projects, you must weigh the cost of investigation against the impact of the failure. If a data gap of a few hours does not affect clinical decisions or supply chain operations, it may be acceptable. I have learned to ask: “Does this failure prevent someone from doing their job, or does it just annoy me?” If it is the latter, I document it and move on.
This is not cynicism; it is triage. When you have one engineer supporting fifty health posts across three regions, you cannot chase every ghost. You fix the failures that harm users, and you build resilience so that the remaining failures are mere inconveniences.
FAQ
What is the most common cause of intermittent failures in off-grid deployments?
Power instability. Voltage fluctuations, battery depletion, and generator cycling cause more intermittent failures than software bugs. Devices reset, corrupt data, or enter undefined states. Always check power first—measure voltage at the device, not at the source, and log it if possible.
How do you debug a failure that you cannot reproduce?
Focus on environmental reproduction, not just code paths. Use a variable power supply, network emulator, and temperature chamber (or a heat gun and a freezer) to simulate field conditions. Collect state dumps and flight recorder data from the affected device. Talk to the user or field officer about what was happening around them—not just on the screen.
Is it worth building custom observability tools instead of using SaaS platforms?
In my experience, yes, if your deployment is in a bandwidth-constrained or budget-constrained environment. Open-source tools like Prometheus, Loki, and Grafana can be self-hosted cheaply. The real cost is the engineering time to integrate them. But that investment pays off because you build exactly what you need and avoid vendor lock-in. You also gain a deeper understanding of your system’s behavior.
How do you convince stakeholders to invest in debuggability?
Frame it in terms of operational cost. Every hour a field officer spends troubleshooting a device is an hour not spent on their actual work. Every site visit to diagnose a problem costs transport, per diem, and lost time. Debuggability features—state dumps, flight recorders, heartbeats—reduce these costs. Show them data from past incidents: how long it took to resolve, what it cost, and how debuggability would have shortened it.
What Comes Next
This article focused on debugging intermittent failures. A natural next step is to explore how to design systems that are resilient to these failures in the first place—patterns like circuit breakers, retry with backoff, and event sourcing that work when connectivity is a luxury. I will cover that in a future piece, with concrete examples from field deployments. If you have your own war stories or techniques, I would like to hear them. The best debugging tool is a community of practitioners who share what actually works when the power is out and the budget is gone.