Your Automation Is a Guest on Someone Else's Origin

The explorer behind infinite-craft.phenomsec.com is one Lambda function on a rate(4 hours) EventBridge schedule, and every pulse calls a single endpoint on a server I do not own. Its pulses started ending with the delay pinned at the 60-second ceiling of its own backoff. Nothing had crashed. The endpoint had moved behind Cloudflare's bot protection, the worker was reading that as rate limiting, widening the delay each time, and then retrying, for the rest of its fourteen-minute budget, a request that no plain HTTP client can complete.

Fixing the classification was correct and immediately made things worse. A verification pulse after the change made 201 API calls with 70 of them challenged, against 91 in the run before it. Better labels had been read as permission to push harder.

Follow one refused request through

The worker picks a pair, say Steam + Cloud, claims it with a conditional write in DynamoDB so no concurrent pulse duplicates the call, and requests https://neal.fun/api/infinite-craft/pair with Accept: application/json, text/plain, */*. Cloudflare, sitting in front of the origin, decides this request gets a managed challenge and returns 403. Python's urllib raises HTTPError. The handler catches it, sees e.code == 403, and returns {"_rate_limited": True, "_status_code": 403}. The rate limiter, an additive-increase/multiplicative-decrease (AIMD) loop, reads that flag, increments consecutive_failures, doubles delay toward its 60-second cap, and carries on around the loop.

Two facts about that response never reached the limiter. Cloudflare sets cf-mitigated: challenge on every challenge page it serves, and its documentation states that challenge is the only valid value that header takes. Cloudflare also documents that a challenge response carries content-type: text/html regardless of the resource type the client asked for. So the worker requested JSON, received HTML and a header naming the mitigation, and discarded both.

The information is lost at exactly one hop: a single line in the except urllib.error.HTTPError block that mapped two status codes onto one boolean. Everything downstream of that line was reasoning correctly about a fact that was no longer true.

A 403 is two different refusals

RFC 9110 section 15.5.4 defines the code this way: "The 403 (Forbidden) status code indicates that the server understood the request but refuses to fulfill it. A server that wishes to make public why the request has been forbidden can describe that reason in the response content (if any)." The reason lives in the content. The content of a challenge page is HTML wrapping a JavaScript challenge, which is the origin saying prove you are a browser. A client that parses JSON will never read that sentence.

The same section is careful about scope. Its SHOULD NOT on automatic retries governs repeating rejected credentials, and says nothing about a client that presented none. Nothing in the specification tells this worker what to do with a bare 403.

429 is not in RFC 9110 at all. It comes from RFC 6585 section 4, which defines it as "the user has sent too many requests in a given amount of time", and the same document's security considerations add a line worth keeping in mind: "servers are not required to use the 429 status code; when limiting resource usage, it may be more appropriate to just drop connections, or take other steps." The informative status code is a courtesy the origin may decline to extend. A taxonomy of refusal built only out of status codes will therefore be wrong on the cases that matter most.

Here is the taxonomy the worker uses now, and what each row licenses:

Signal	What the origin is saying	Correct response
429 or 403 carrying Retry-After	You are going too fast	Wait the value given, clamped to 300s
403 with cf-mitigated: challenge	You are the wrong kind of client	Back off; stand down if it persists
403 with neither	Undiagnosed refusal	Back off; assume the challenge case
Connection reset, no response	You have no signal at all	Back off; you cannot reason about this

Every row's response is a smaller number of requests. Row two is the one that gets handled wrong, because it is the only row where waiting genuinely cannot fix the problem, which makes waiting feel pointless. That is the trap I walked into.

The fix that doubled the traffic

The first attempt was right about the classification and wrong about what to do with it. It split the challenge out of the rate-limit branch, ended the pulse after CHALLENGE_ABORT_STREAK (set to 5) consecutive challenges, and dropped the AIMD backoff on the challenge path entirely. The reasoning was defensible: a delay cannot clear a JavaScript challenge, so waiting before the next attempt buys nothing.

Waiting buys nothing for me. It buys something for the origin.

Challenges turned out to be intermittent rather than sustained. Its commit message records a recent run getting 85 of 91 calls through. A streak of five in a row is therefore rare, so the abort almost never fired, and with the throttle removed the worker ran at full speed through a stream of refusals. The commit that repaired it records the verification pulse that followed: 201 API calls with 70 challenged, roughly 35 percent, against 91 calls in the run before the change. I had removed a throttle that was firing for the wrong reason and put nothing in place that fired for the right one.

The repair was seven lines:

# Back off exactly as for a rate limit. Backoff cannot clear a
# JS challenge, but a challenge still means the origin is
# refusing this traffic, and the correct answer to that is to
# send less of it. Skipping the backoff here (an earlier version
# of this branch did) doubled request volume against a server
# already turning a third of it away.
delay = min(60, max(known_safe_delay, delay) * 2.0)

What changed is not the classifier, which was already correct. What changed is that both refusal branches in the pulse loop now widen the delay before either of them decides whether this particular refusal is worth continuing past. Handling follows from what the response says about the origin's state, not from what it does for you.

Three ways to finish a run used to look identical

A pulse that emptied its frontier, one that hit its Lambda deadline, and one turned away by Cloudflare all used to write the same row into the worker_runs audit table: full duration, zero discoveries, no explanation. Three unrelated conditions produced one indistinguishable record, which is why I cannot tell you when the bot protection first appeared.

Each run now records challenges as a count and stop_reason as one of completed, frontier_exhausted, time_exhausted, or cloudflare_challenge. That is a two-field change and it is the reason the previous section contains "70 of 201" rather than "it seemed worse." Governing a system that runs without you is mostly the ability to answer why did it stop from a table rather than logs, because nobody reads logs for a run that did not look like a failure.

Send fewer requests by measuring which ones pay

Rate discipline is usually discussed as pacing. There is a second lever: send fewer requests in total by not sending the ones that have never produced anything, which means weighting the choice of strategy by what each one has returned per call. The worker_runs history was blunt about which was not paying. The bfs strategy had recorded 0 discoveries across 172 runs; random had recorded 7 across 9. Uniform selection was spending a third of every pulse on the arm with a measured yield of zero.

The history was blunter than it was fair. That 172-against-9 imbalance is itself a bug's signature: a fix from a week earlier records that strategy rotation had been frozen on bfs and that bfs was re-proposing pairs it had already tried, so it burned whole pulses without making a call. Weighting by a yield measured while an arm was broken buys a rate reduction now and a distorted prior later.

Strategy selection is now weighted by discoveries per API call, with a 10 percent floor so a cold strategy keeps being sampled and can recover, and a 50-call minimum below which a strategy is scored at the average of the measured ones instead of at zero. Applied to the history above, that moves bfs to roughly 10 percent of pulses and random to roughly 57 percent. The gain is not that the explorer discovers more. It is that the same discoveries now cost someone else's origin fewer requests.

The strongest objection is the User-Agent

The strongest objection to all of this is that the worker is not much of a guest. Its request headers are a browser's, down to the client hints. That is a Lambda function presenting itself as a browser on somebody's desk. On that reading the challenge is not a signal I misread; it is the origin correctly refusing to distinguish a client that went out of its way to be indistinguishable. My taxonomy, my backoff and my stop reasons are careful conduct inside a posture that was already dishonest.

That objection is right, and I have not fixed it. The headers were written before any bot protection existed on that endpoint, and were never revisited. Once a challenge appeared, the honest response was to change how the client identifies itself, not only how fast it goes. What I would defend is that the two are separable, because a client that lies about its identity and sends 91 requests still costs the origin less than one that lies and sends 201. What I concede is that identification is the larger half of the fix and I did the smaller half first, because it was the half my instrumentation could see.

One thing does not settle this. neal.fun/robots.txt is 37 bytes long: one Sitemap: line, no User-agent, no Disallow. The absence of a prohibition is not a grant of permission, and an API endpoint was never in scope for that file anyway. The challenge is the permission signal, and it says no.

What I still cannot answer

The yield weighting has a floor and no ceiling worth the name. With three arms at a 10 percent floor the winner is capped at 80 percent, which is not a constraint that binds. If exactly one strategy has a non-zero measured rate it takes nearly all of the probability mass above the floor, and a starved arm stops producing the measurements that would show it had recovered as the frontier shifted. That is the same trap as weighting on a rate a bug produced, one step further out. I do not know whether the fix is a fixed ceiling or a blend that starts uniform and slides toward measured yield as the thinnest arm fills in.

In the first post in this series I listed "design for rate discipline" as one of five principles and gave it three sentences, one of which was that your agents are guests on other people's infrastructure. Its checklist, one bullet further down, asks for graceful degradation: "When the external service is unavailable, the system needs to fail safely rather than retry aggressively." Those sentences did not prevent any of this, because a principle does not contain the operational rule.

The rule is this: when a classification improvement raises your request volume, the classification is not the bug, the objective is. It holds specifically where you are the client rather than the origin. If you own both ends, better classification genuinely should let you push harder, and that asymmetry is the entire point.