Building a Reliable Filesystem for Embedded Gateways: Lessons from the PoE GW Project

When working on embedded systems that run unattended in the field, powering sensors, cameras, or industrial devices, one silent failure can be costly. A corrupted filesystem or bad update shouldn’t require a technician with a USB stick to fix it.

That’s the philosophy that guided our design for the Siggate-PoE Gateway (PoE GW), an industrial PoE-powered edge gateway running a Yocto-based Linux distribution on a TI Sitara (AM335x) platform.

Designing for Reliability: The Filesystem Layout

At the heart of the PoE GW design is a ext4-based overlay filesystem that cleanly separates the immutable system image from persistent user data.

Our partition scheme looks like this:

This structure gives us the best of both worlds:

• A read-only base system that’s resilient against corruption.
• A persistent writable layer for configuration, logs, and runtime data.
• A clear recovery path when something goes wrong.

OverlayFS: A Smart Way to Stay Writable

Instead of keeping the entire root filesystem writable (and vulnerable), the PoE GW boots into a merged root built on top of two layers:

lowerdir=/sysroot      # Read-only base
upperdir=/var/upper    # Writable overlay
workdir=/var/work      # Overlay workspace
merged=/mnt/merged     # Mount safely to a temp path, then pivot the root to it

The system mounts these into /sysroot-overlay and pivots root during early boot.
This design means:

• OS Protection: Core system files remain completely read-only and immune to corruption.
• Isolated Changes: All modifications, logs, and updated configurations are safely isolated in /var/upper.
• Atomic Integrity: /var/work serves as an internal kernel scratch space to prevent data corruption during writes.
• Unified Presentation: /sysroot-overlay serves as the final combined folder where the system reads and writes files.
• Instant Recovery: Clearing the /var/upper directory instantly resets the device back to its factory state.

Updates and Recovery

Updates are handled as atomic replacements of the base rootfs image.
If an update fails or the overlay becomes inconsistent:

• The system automatically reverts to a clean read-only root.
• The LED signals a fault state.
• Recovery tools in the initramfs allow for remote or manual repair.

Future plans include A/B rootfs subvolumes using ext4 snapshots — enabling rollback with a single boot switch.

Persistent Identity and Provisioning

Each gateway carries unique data stored in a tiny raw EEPROM partition:

• Serial number
• Product number
• Manufacturing revision

Our provisioning script writes and verifies this data at the factory, communicates with our backend database, and even prints a manufacturing label through a Zebra printer, all while ensuring the EEPROM remains consistent and locked.

Why This Matters

In field deployments, a stable filesystem is more than convenience, it’s trust:

• No more corrupted eMMC file systems after power loss.
• Predictable behavior after every update.
• A clear line between system integrity and customer configuration.

When you have thousands of gateways deployed across industrial sites, this design saves real engineering time, and builds confidence in the product.

Final Thought

Reliability isn’t an accident. It’s an architecture choice.
By layering simplicity with safety, we’ve built a PoE gateway that can take a beating and keep running... exactly what you want from an industrial device you may never see again after it ships.