Embedded software doesn’t operate in isolation. It depends on processors, sensors, memory, communication interfaces, power systems, actuators, and other hardware to do its job. That creates an uncomfortable question for any system that needs to remain useful when something goes wrong: What should the software do when the hardware it depends on stops behaving normally?
The answer isn’t always “restart.”
In a well-designed system, software can detect certain abnormal conditions, determine what they mean, isolate the affected function when possible, and move the system into a state that limits further problems. For embedded software companies, designing that response is part of making a system dependable under real operating conditions.
First, the Software Has to Recognize Something Is Wrong
A failed hardware component doesn’t always announce itself clearly. A sensor might stop producing data. It might produce a value that is technically readable but physically impossible. A communication device might stop responding. A peripheral might become unavailable while the processor itself continues running.
The software therefore needs some way to distinguish normal behavior from an abnormal condition. That can involve timeouts, status checks, range checks, communication monitoring, diagnostic information, or other fault-detection mechanisms. The exact method depends on the hardware and the function being monitored.
The important point is that recovery can’t begin until the system has some reasonable way to detect the fault.
Detection Is Only the Beginning
Once a problem is detected, the next question is harder: What should happen now?
Suppose a noncritical sensor stops responding. The system might be able to continue using another source of information. It might mark the affected data as invalid. It might retry communication. It might reduce functionality while continuing with the remaining functions.
Now imagine that the failed component controls something essential. The response may need to be very different. NASA describes fault-protection systems that can move spacecraft into protective or “safe” modes when certain failures occur. Those responses can include actions intended to preserve the spacecraft and help recover communication or control.
The lesson is straightforward: recovery behavior needs to be designed around the consequences of the failure.
A Restart Doesn’t Fix Every Fault
Restarting software can be useful, but it isn’t a universal recovery strategy. If the underlying hardware problem remains, a restart may simply reproduce the same failure. There is another problem. A system can sometimes continue running while producing incorrect results rather than stopping altogether.
NASA research examining historical aerospace software failures found that erroneous outputs were more common in its dataset than simple no-output failures. The study also found that rebooting was not reliably effective for clearing erroneous behavior.
That makes fault handling more complicated than: Something went wrong. Reboot it.
Engineers need to consider what caused the abnormal condition and what state the system will be in when it comes back.
Fault Isolation Keeps One Problem From Spreading
Imagine a system with several independent functions. If one peripheral fails, the software shouldn’t necessarily treat the entire product as failed. Fault isolation is about separating the affected function from the rest of the system where that is technically and operationally possible. The approach depends heavily on the architecture.
A system may disable one function, switch to redundant hardware, use a backup data source, or move into a reduced-capability mode. NASA’s spacecraft fault-protection examples include switching to redundant hardware as one possible response to a component failure.
That doesn’t mean redundancy is appropriate for every product. Additional hardware brings its own cost, complexity, weight, power, and failure considerations. The point is to decide what level of isolation the mission actually requires.
Safe States Need to Be Defined Before a Failure
There is no single “safe state” for every embedded system.
- For one product, it might mean shutting down an actuator.
- For another, it could mean maintaining a controlled operating mode with reduced functionality.
- For a vehicle, it might mean preserving control while disabling a nonessential subsystem.
The right response depends on what the system is doing and what could happen if it continues operating incorrectly.
NASA’s software engineering guidance includes fault detection, isolation, and recovery as part of software safety analysis. Its documentation also illustrates the use of watchdog mechanisms to detect certain processor failures. The key decision is made during design, not after the first hardware failure occurs.
Watchdogs Can Help With Certain Failures
A watchdog timer is one familiar example of hardware or software monitoring. The basic idea is simple. A running system periodically provides an expected signal or “heartbeat.” If that signal stops arriving, the watchdog can take a predefined action, such as resetting the processor.
NASA’s aerospace lessons-learned material describes hardware watchdogs as an independent fault-protection measure for real-time systems. But even a watchdog has limits.
It may detect that a processor has stopped responding, but it doesn’t automatically understand why. It may not solve a faulty sensor, damaged communication path, incorrect data, or a hardware condition that leaves the processor running while the system behaves incorrectly. Good fault handling therefore uses the tools that match the failure modes being considered.
Recovery Should Leave Useful Evidence Behind
A system that recovers silently can be difficult to diagnose later. If a component fails and the software simply restarts, engineers may have little information about what happened. Logging can help capture useful details such as the fault condition, affected subsystem, timing, error state and recovery action.
The amount and type of information depend on the system. In a constrained embedded device, storage and processing resources may limit what can be recorded. In a larger system, more extensive diagnostics may be possible. The goal isn’t to save every possible piece of information. It’s to preserve the information that can help determine whether the recovery worked and whether the underlying problem needs attention.
Testing Has to Include the Bad Days
A recovery strategy that exists only on a design document hasn’t proved much. Engineers can test how software behaves when communication is interrupted, sensors return invalid information, components stop responding or other defined fault conditions occur.
NASA’s space-systems lessons emphasize realistic nominal and anomalous scenarios and hardware-in-the-loop testing for critical flight software functions. Testing can also reveal an uncomfortable result: the planned recovery may create a new problem.
For example, a reset might occur at an unsafe moment. A fallback mode might consume more power than expected. A repeated retry could prevent the system from moving on. A watchdog could restart a system without fixing the condition that caused the failure. Those are exactly the kinds of problems worth finding before deployment.
Hardware and Software Have to Be Considered Together
A software team can’t define every recovery behavior without understanding the hardware architecture. Likewise, hardware decisions can affect what software can detect, isolate, or recover from.
At Swift Engineering, we approach embedded development with that connection in mind. Our work has to account for how the software interacts with the physical system rather than treating code as a separate layer that can be designed without regard to the hardware beneath it. That becomes particularly important when the product has to operate with limited access to maintenance or when an unexpected failure could affect the larger mission.
Final Thought
For embedded software companies, dependable software isn’t only about making normal operation work correctly. It also means deciding what the system should do when a hardware component doesn’t. That requires more than a restart command.
The system may need to detect the problem, isolate the affected function, enter an appropriate operating state, preserve useful diagnostic information, or use available redundancy. The right combination depends on the product and the consequences of failure.
At Swift Engineering, our goal is to make those decisions part of the engineering work rather than something left for the first real failure to reveal.
