What Dies First When You Retire a Service

for id in nfd32ilbk4 ufgv1pgb5b; do
  aws apigateway delete-rest-api --rest-api-id "$id" && echo "  deleted $id"
  sleep 35   # API Gateway throttles deletes hard
done

That is step 4 of eight in the script that took down aitc.phenomsec.com, a demo subdomain that tracked how large language model providers change their legal terms and never found users. The sleep 35 is not superstition. AWS publishes a fixed quota of one DeleteRestApi request every 30 seconds per account and states that it cannot be increased, so two REST APIs take at least half a minute longer than anyone estimates, and a script that fires them back to back throttles on the second call and stops in the middle of a destruction.

Retirement is the one infrastructure operation that cannot be retried. Anything else you run against an account can be run again if it goes badly; once the buckets are gone there is no second attempt. That single property is what makes ordering, and the evidence you take before you begin, the entire job.

What I expected

I expected the hard part to be finding everything. The service had Terraform in the repository and a console I could walk, so I assumed the risk was an orphaned resource billing quietly for a year. That turned out to be the easy part. Counted from the eight steps of the teardown script, the list came to a CloudFront distribution, four EventBridge rules, eight Lambda functions, two API Gateway REST APIs, five DynamoDB tables, six S3 buckets, four IAM roles with three customer-managed policies, and one certificate in AWS Certificate Manager (ACM).

The three things that actually cost time were not on that list. Proving the archive was complete before I held the ability to destroy the original. The order, which the platform dictates and you discover by violating it. And everything still pointing at the hostname, which no inventory held.

Preservation is a gate, not a step

The service's value was never its code. It was a corpus of provider legal documents captured over time, and the local archive directory holding it now carries 15,238 objects and 5,123,020,208 bytes of it, counted with find against the local copy after the download. Alongside it sits a 57 MB git bundle carrying every branch, and the index that makes the corpus navigable: 14 providers, 66 documents and 633 versions, counted from the fourteen tracked versions.json files in the repository.

The number that made this a gate rather than a chore is inside that 633. Of those versions, 131 carry "source": "archive.org", meaning they were recovered from the Wayback Machine after the providers replaced those pages. Those 131 cannot be produced again from any live source. Nothing about deleting a DynamoDB table warns you that you are also deleting the only organized copy of seventeen years of terms-of-service revisions, the earliest of them from 2008.

A commit landed on the project the same day it was retired, titled "Record that production reads dev tables and has lost its version history." The site that served the corpus had been publishing an empty index. Had I archived what the live site served rather than what the repository and the buckets held, I would have preserved an empty file and destroyed the real record in the same afternoon.

So the sequence is fixed and it has an owner: whoever runs the destruction proves the archive is complete and readable before anyone grants them delete permissions. Doing it in that order costs a day. Doing it in the other order is unrecoverable, and you find out weeks later when someone asks a question the archive cannot answer.

The order is set by the platform, not by you

Four of the eight steps could not be moved, and each is pinned by something outside my control:

Resource	What forces its position	What happens if you move it
Cloudflare DNS record	Must go first, before any AWS work	The name resolves to a Cloudflare error page once the origin is gone
CloudFront distribution	Disable, wait for Deployed, then delete	DistributionNotDisabled, HTTP 409
API Gateway REST APIs	One delete per 30 seconds, account-wide	The second call throttles and the script stops mid-teardown
ACM certificate	Only after the distribution is deleted	The certificate is still in use and the delete is refused

Row one is the one that gets missed, because it is the only row that is not in AWS. The DNS record was not in the teardown script at all. The local Cloudflare token carried zone:read and nothing else, so the script's header carries the instruction instead: remove the record in the dashboard FIRST, or the name resolves to a Cloudflare error once the origin is gone. A pointer that outlives its target is worse than a name that stops resolving, because an error page looks like an outage and produces a support question rather than an obituary.

Row four has a second trap that only appears when you run it. The ACM certificate lives in us-east-1 wherever the rest of the stack runs, and the script's own execution note records that steps two through five, seven and eight all ran as written with a single exception: step eight needed the ACM permission on the elevated role, not on the administrative user. A missing grant surfaced on the last line of an eight-step destruction, which is the worst place in the sequence to find one.

The delete step in my own script was wrong

Step 6 walked the six versioned buckets, five of which still held versions, and I wrote it the obvious way: page through list_object_versions with a paginator and delete each page's versions inside the loop. It is the shape every example uses. It ran for roughly 25 minutes against one bucket, aitc-website-dev, without finishing, and I killed it.

Deleting objects while paginating the listing of those same objects makes the paginator thrash: the continuation state describes a listing you are actively destroying, so the walk keeps restarting against a shifting set. The replacement, kept as _archive/s3-purge-versioned.py, does the one thing the loop refused to do, which is start over:

while True:
    r = s3.list_object_versions(Bucket=bucket, MaxKeys=1000)
    objs = [{'Key': o['Key'], 'VersionId': o['VersionId']}
            for k in ('Versions', 'DeleteMarkers') for o in r.get(k, [])]
    if not objs:
        return
    yield objs

Re-listing from the top after every batch converges, because each pass sees a strictly smaller bucket. With eight worker threads it cleared 40,669 versions and delete markers in about 25 seconds, according to the note I left in the script: 39,606 of them on aitc-documents-dev, the rest split across three smaller buckets. The MaxKeys=1000 is not a tuning choice: the S3 API reference caps a DeleteObjects request at 1,000 keys, so the page size and the batch size are the same number by definition.

What I changed permanently is smaller than the script: the teardown file now opens with a note saying step 6 as written is wrong and must be fixed before anyone reuses it.

Nothing tells the consumers

The orphaned pointer. Every reference to a service is created in a moment of enthusiasm and stored somewhere that has no idea what it points at, which is why retirement breaks things that were never part of the system.

The evening the resources came down, one commit on the website removed 83 lines from src/app/resources/page.tsx. It deleted an entire "AI Compliance & Terms Tracking" section: three entries, two of them linking to aitc.phenomsec.com, the third pointing at a /blog#ai-compliance anchor the rebuilt blog no longer has. Under them sat a call to action with a third link to the same dead host:

Bookmark the tracker and get real-time visibility into how your AI vendors update their legal terms.

Open AI T&C Tracker →

"Real-time visibility" is the phrase that does the damage: a freshness promise still rendering for anyone who loaded the page, pointing at a hostname that had stopped resolving hours earlier. The same sweep turned up three "Frameworks & Guides" entries that had never carried an href at all, each rendering a full paragraph of description above the words "Coming Soon", and two posts from 2020 linking to /contact, a route this site has never had. None of it was found by a scanner. It was found by opening the page and clicking.

Credentials are the residue that lasts longest, because nothing exercises them until someone does. Three profiles naming the retired service are still in ~/.aws/config. One of them assumes the role that step 7 deleted, and it chains from a source_profile that exists nowhere else in the file, so it was already broken before the teardown and nobody noticed: a profile is only evaluated when a human types its name. That is why an inventory built by reading configuration files reports roles that do not exist.

The strongest objection

The strongest objection is that this is a demo subdomain with one operator, and that eight hand-ordered steps and a five-gigabyte archive are ceremony that will not survive contact with an organization retiring twenty services a year. Dependency order is exactly what infrastructure-as-code exists to solve. terraform destroy already knows the certificate depends on the distribution, so hand-writing the order reinvents something the tool does correctly.

That is right about ordering, and I would use terraform destroy first on anything whose resources are genuinely all in state. It is wrong about the two things that cost the most. The Cloudflare record was a different provider under a different token and was never in the AWS state file, and every consumer of the hostname was outside every state file by definition. The tool handles the dependency graph. It does not handle the preservation gate, and it cannot know what points at you.

A retirement you could run this week

If you have a service you know is finished, this is the version that fits in a day:

1. Build the resource list twice, once from state and once from the console, and diff them.
2. Copy the data out, then count objects and bytes on both sides and compare the totals.
3. Open one archived file of each type and read it before anyone is granted delete permissions.
4. Grep every repository, wiki page and ~/.aws/config you own for the hostname and the resource names.
5. Remove the DNS record first, then work inward from the edge.
6. Annotate the script with what actually happened, including what failed, and keep it.

A bad result is step 4 returning hits you cannot explain. Every one of those is a consumer you were about to break without knowing, and finding three of them is a much better outcome than finding none.

The rule I take from this is that the last artifact a retired service produces is the script that killed it, annotated with everything that did not go as written. Nobody can inspect that CloudFront distribution or those buckets now to settle an argument about what they held. They can read that file, which is worth more than the runbook that preceded it, because the runbook records what was supposed to happen and the annotation records what did.