Edge Failure Modes Are Network Failure Modes First

Most edge incidents start the same way. Grafana goes empty. The GitOps agent stops reporting. Someone concludes the NVIDIA Jetson is down, the model crashed, or K3s ate itself. A site visit later, the camera pipeline is still producing boxes. The node never crashed. The path to it did.

Cloud GPU clusters train teams to treat silence as death. If Prometheus cannot scrape, the replica is gone. If the API server cannot schedule, the workload is gone. That heuristic is cheap in a datacenter. It is expensive in a factory, a store, or a vehicle, where the first thing to fail is almost never TensorRT or GPU memory. It is reachability.

Edge failure modes are network failure modes first. Until the platform can distinguish a partition from a crash, every blank panel becomes an outage, and every outage becomes a truck roll.

A crash means the inference process, the node, or the device has actually stopped doing useful work. A partition means the control plane and the site cannot talk. Those are different systems with different remedies. Collapsing them into one alert is how teams burn nights on the wrong problem.

On a well-built edge node, serving does not require a live round trip to the cloud. The last pinned image and model package are local. Health checks can run on-device. The business outcome, occupancy, defect flags, whatever the model is for, can continue while Git cannot apply, the registry cannot serve layers, and the VPN cannot keep a scrape path open.

The operator experience is inverted. The control plane looks dead. The site may be fine. Designing for that inversion is the difference between an edge demo and an edge platform.

Hardware catalogs emphasize TOPS and camera interfaces. Field operations spend their time on links that flap, NAT that changes, and firewalls that were fine until someone rotated a certificate.

Cloud-native tooling reports absence. Absence is not the same as failure. A missing scrape sample, a stale Fleet report, or a red Kubernetes condition often means the observer lost the node, not that the node lost the workload.

The second trap is treating SSH as truth. If a tunnel is up, the site is healthy. If it is not, the site is down. That collapses identity, overlay, and application health into one boolean. It also trains the team to wait for connectivity before they can even name the incident.

The third trap is a single SLO copied from cloud serving: availability equals successful remote checks. At the edge, availability has to be defined where the work happens. If the model is still inferring on-device, the customer-facing system may be up while your platform UI is dark. Those are two different error budgets.

You do not fix this with a louder VPN. You fix it by making network partitions a designed state, the way Kubernetes already treats node NotReady, except you cannot assume the control plane will be there to record it.

An on-call engineer should be able to answer three questions without guessing: is the site unreachable, is the workload actually down, and which release is it running. If those collapse into one red tile, the investigation path is improvisation.

Useful remote signals are boring on purpose. Time since last heartbeat. Time since last GitOps apply. Image digest on the node versus digest in Git. Local restart counts buffered until the link returns. Temperature and disk pressure if you have them, because they explain crashes that networks did not cause.

When the link is gone, the node still needs a story. Persist a short local log. Keep the previous package. Make last-known-good the default, not an afterthought. The control plane should reconcile when the partition heals, not require a human to reconstruct what happened from memory.

Cloud canary often means a percentage of traffic. Edge canary means a cohort that can survive being cut off: a lab bench, one store, one device class. You prove the release where you can still afford a site visit, then you let the rest pull when they can.

Artifact size is a network problem wearing a DevOps badge. A model that is trivial to pull in a region is a failed rollout on LTE. Local caches, scheduled windows, and keeping the previous package on disk are not optimizations. They are how rollback remains possible after the WAN dies mid-apply.

GitOps still belongs here. The mistake is pretending the apply loop has datacenter latency. Desired state can live in Git. The fleet will converge later. Operators need to see lag as a condition, not as proof that GitOps does not work at the edge.

If you already run inference on NVIDIA Jetson or a similar class of device, stop treating blank metrics as a crash until you have a heartbeat that is independent of a full scrape path. Add last-applied release next. Then rehearse a partition: unplug the WAN, confirm serving continues, and confirm you can still roll back without the overlay.

If you are still cloud-only, do not wait until the first factory deployment to invent this. The habits transfer: immutable artifacts, pinned releases, and health that names its failure domain. Edge only makes the network failure domain impossible to ignore.

Sailbird builds edge inference fleets with that assumption from the start: K3s on devices such as the NVIDIA Jetson Orin Nano, Git-backed desired state, cohort promotion, and observability that can survive a site going dark. The model is yours. The packet path is part of the platform, not a prerequisite you hope holds.