Fail Closed on the Server, Fail Open at the Client

Two weeks ago I changed a middleware default so that a missing enforcement flag resolves to full enforcement in production. The next day I shipped a gate in which an unknown platform value, an unparseable version string, a missing database row, and a failed database read all let the request through. Same repository, one day apart, and both were right. "Fail closed" is not a property of a well-built system. It is the answer to a question that has to be asked once per control: when this thing is wrong, who gets refused?

Refuse the caller who can retry in a second

Device attestation on the backend of a mobile app I build is controlled by an environment variable, ATTESTATION_ENFORCE_MODE, set per environment in wrangler.toml and read as off, soft, or hard. Until that change an absent value resolved to off, so a dropped or misspelled key produced a production deployment with attestation switched off, returning 200s on every mutation route and looking from the outside exactly like a healthy one. The fix left every explicit setting alone and changed only the absent case: on production a missing value now resolves to hard, and everywhere else it still resolves to off.

The same typo now produces 403s on production writes rather than quiet success. That is a bad morning for whoever is on call, and it is the right bad morning. A false positive here refuses a request whose sender can re-attest and retry within a second. A false negative leaves every route the middleware guards unprotected for as long as nobody thinks to check, and the two outcomes are separated by a single missing line of configuration.

The next day's change went the other direction on purpose, and the reason is that it reaches software I no longer control.

The controls you cannot take back

Migration 0021 adds a table, app_metadata, keyed on (platform, flavor), holding a minimum supported version, a recommended version, a current latest version, and two nullable message columns. It ships seeded for all six platform and flavor pairs at min_supported_version = '2.0.0', which is the oldest shipped client, with both message columns NULL. On the day it deployed it blocked nothing. The rows are edited with wrangler d1 execute, so raising the floor later is an UPDATE statement against one row, not a code change and not a deploy.

The middleware reading that table returns HTTP 426 with the error code CLIENT_VERSION_UNSUPPORTED when a client reports a version below the floor. It does that only on mutation methods, and only when the request carries both X-Client-Version and X-Platform. The comment above it says why:

// ROLLOUT POSTURE — requests WITHOUT the headers are NOT gated. This
// is the designed behavior, not an oversight: v2.0/v2.2 production
// clients predate X-Client-Version/X-Platform and gating header-less
// traffic would brick every one of them the moment this deploys.

Every ambiguous case resolves to pass: an unrecognized platform string, a version that will not parse, a missing row for that pair, and a failed read against the database, which logs a warning and then calls next() anyway. Reads are never gated at all, so a client refused a write can still render the data it already has underneath the upgrade prompt.

One thing about it is wrong on the standard. RFC 9110 §15.5.22 says a server sending 426 MUST include an Upgrade header naming at least one protocol it accepts, because the status is about protocol version. This middleware sends X-Client-Min-Version-Ios and X-Client-Min-Version-Android instead. The clients switch on the error code and never read the status line, so nothing breaks, but the code is borrowed rather than earned, and an intermediary is entitled to be confused by it.

The escape hatch I shipped and then removed

The reason I trust the header-conditional design in that version gate is that I built the same shape into a security control two months earlier, where it was a bypass.

In the spring, device attestation was mounted in hard mode on the mutation routes and the Android client could not attest, because Play Integrity was blocked on a business validation that had not cleared. So the middleware carried an exemption: if the request header X-Platform read android, it called next() and returned. The comment above the block read "Remove this block once PR D is live and Android can provide X-Device-Token", which is the sentence you write when you believe you will.

A pre-production review flagged it as blocking. The header is unauthenticated, so any client at all could send X-Platform: android and skip attestation on every route the middleware guarded. The fix deleted the block, and the comment left in its place is the lesson in one line:

Without a signed platform claim, trusting the header is a bypass, not a policy.

That removal knowingly broke something. The pull request records that an Android tester still running an old development build would now get 403s, and accepts it, because no Android traffic was expected at those routes and the alternative was an open door. Two months later the same header name appears in the version gate and is fine there, for a reason the middleware states directly: lying about your platform or version only ever grants the liar passage, and never blocks anyone else.

Who pays when the control is wrong

The question that decides the direction is not how sensitive the thing behind the door is. It is who is standing in front of the door when the control fires by mistake.

Control	Refused when it is wrong	Direction	Cost accepted
Enforcement flag missing from config	A caller who can re-attest and retry	Closed	A loud outage instead of a silent gap
Version floor for already-shipped clients	Every install of a build you cannot patch	Open, seeded inert	No teeth until an operator gives it some
Pinned Apple App Attest root certificate	Every iOS device at once	Closed	Apple rotating the root takes attestation down
Unblock call naming an unknown user id	Nobody, because the answer is the same either way	Neither	A pointless delete, in exchange for no existence oracle

Row three is the one that gets waved through. Pinning a trust root feels like an unambiguous fail-closed decision, and it is, but its false positive is total rather than per-user. The pinned fingerprint is checked on the fetch and on every cache read alike. If Apple rotates that root, every App Attest verification fails until someone edits a string constant and redeploys. That was accepted with the remediation written in the comment beside the constant and the certificate's own expiry date, recorded in that same comment and nearly twenty years out, as the argument that the day is distant.

Row four is the same question asked about information rather than access. The unblock endpoint deletes a row that may not exist and answers 200 either way, because the only party a strict existence check would inform is someone probing which account identifiers are real. The PUT on the same resource does return 404 for a target that does not exist, which is the same oracle the DELETE refuses to be. The asymmetry is deliberate, and the PUT is the weaker half of it.

The strongest objection

The strongest objection is that a control which fails open is not a control, it is a log line. The version gate is defeated by omitting a header. The platform value is client-asserted and unauthenticated. If the database read fails the gate disappears for the duration, and the only trace is a warning nobody is paged for. Ship six permissive rows plus a code path that concedes on every ambiguity and you have built a dial that reads zero, which is not distinguishable from a piece of wire.

That is right, and the code says so in writing: "this is a UX gate, not a security boundary." The further concession I will make is that the fail-open on a database error is the weakest part of the design, because it is the only case where the gate stops working for a reason nobody chose. Where the objection goes wrong is in assuming the gate was ever the security control on those routes. Device attestation is, on the same routes, and it fails closed. The version gate exists to retire clients whose server contract has moved. A mechanism that can only ever be applied to callers who volunteered to identify themselves is a poor security boundary and a perfectly good compatibility contract.

What would change my mind

The design rests on a claim I have not tested yet: that gating only header-carrying clients is a rollout stage rather than a permanent hole. One measurement settles it. The first time the floor is raised above 2.0.0 on production, the count of 426 responses should track the share of installs that the app stores' own version distribution reports below that line. If those two numbers disagree by much, the gate is measuring self-reporting rather than the fleet, and header-less traffic needs a decision of its own instead of a default.

Until then, the rule the two commits argue for is this. Name the party who is refused when the control fires wrongly, before you choose which way it fails. If that party is an attacker, or a caller who can retry in a second, fail closed and make the failure loud enough to page someone. If it is an install you cannot reach, ship the dial seeded at zero and turn it later with an UPDATE statement and a number, never with a deploy.