Canary Is a Cohort, Not a Percentage

Cloud canary culture is built on a load balancer. Send 5% of traffic to the new revision. Watch error rate. Promote or abort. The unit of risk is a request, and requests are cheap to split.

Copy that habit onto an NVIDIA Jetson fleet and the math becomes theatre. Five percent of twelve devices is not a sample. It is a rounding error, or a single board that happens to sit in the lab. Traffic percentages also assume the next request can land on any replica. At the edge, the next frame belongs to one camera on one node in one building. There is no balancer to hide behind.

Canary is still the right idea: prove a release on a limited blast radius before everyone inherits it. The unit has to change. At the edge, canary is a cohort. Membership is a label, not a percentage.

A GPU serving cluster is usually homogeneous enough that one replica is a fair draw from the rest. Nodes share an image, a SKU, a network, and a region. The control plane can shift traffic in seconds. A bad revision is visible in the same dashboards you already trust, because the canary and the baseline share a failure domain.

That is a gift. It is also a set of assumptions. When they hold, percentage rollouts are honest. When they do not, the same YAML still deploys, and the word canary keeps getting used for something that no longer measures risk.

Edge fleets are small, lumpy, and locally coupled. A store's three devices share a WAN, a switch, and a power circuit. A 'random 10%' that happens to pick all three has not limited blast radius. It has concentrated it. A 'random 10%' that picks three devices in three countries has not tested the site that actually matters this week.

A cohort is a set of devices that share a risk profile you are willing to burn together. That sounds blunt because it is. You choose the grouping on purpose: lab bench, one canary site, one hardware SKU, one network class, one customer who agreed to go first.

Membership should live in Git as labels, not in a spreadsheet and not in whoever answered Slack. site=lab, hw=orin-nano, risk=canary, network=lte. Promotion is then a label change or an overlay, not an SSH loop and not a kubectl patch that means 5% if you squint.

The test of a good cohort is operational, not aesthetic. Can you watch it as a group. Can you freeze it. Can you roll it back without touching the rest of the fleet. If you cannot name who is in and who is out, you do not have a canary. You have hope.

Cohorts fail when they become a schedule. Monday lab, Tuesday store, Wednesday everyone. A calendar is not a gate. The canary cohort has to become boring first: inference errors, restarts, thermal headroom, sync lag, and whatever business check actually matters at that site.

Hold the next group until those signals are dull. That is slower than a percentage slider. It is also the point. Edge blast radius is measured in buildings and production lines, not in a 15-minute error-rate chart that never saw the LTE site.

Skip a cohort only when the risk is truly identical. Identical is rarer than teams admit. A lab device with a perfect power supply is not a canary for a ceiling node that throttles at 2 p.m.

If promotion is membership, rollback is membership too. Pin the canary label to the previous digest. Do not invent a special field procedure because 'it is only three devices.' Three devices that share a site are the entire site.

Keep the previous package on the node. A cohort that cannot move backward without a registry pull is not a canary under partition. The last post in this series is about that failure mode. Canary policy has to assume the WAN may leave during the window you most want to reverse.

Fleet-wide is a destination after the named groups have been uneventful. It is not a first step with a 5% story attached for comfort.

Cohorts are not an excuse to abandon GitOps. Desired state still lives in Git. The overlay is which label selector a release binds to. Cloud GPU serving can keep percentage or replica canaries where the balancer is real. Edge uses selectors. The language for humans should stay the same: prove, then promote, then roll back the same way you rolled forward.

What should not stay the same is pretending Kubernetes traffic splitting is happening on a device that owns its own camera. That mismatch is how 'we canaried it' appears in a postmortem for a fleet-wide bad model.

Pick three cohorts and write them down: lab, one real site, everyone else. Label the devices. Promote a harmless config change through that path before you promote a model. Watch whether you can answer, at any moment, which digest each cohort is running.

If you cannot label devices yet, you are not ready for canary anything. You are ready for inventory. Percentage rollouts will not substitute for that.

Sailbird treats cohort promotion as part of the platform, not a later optimization: K3s fleets on hardware such as the NVIDIA Jetson Orin Nano, Git-backed selectors, health gates between groups, and rollback that does not wait for a perfect network. The model is yours. The blast radius has to be a group you can name.