Freeze, Destroy, and the Grants You Give Back

Over about three weeks last month I shut down two systems living in the same AWS account. One was frozen: infrastructure destroyed, artifacts kept so the platform could be rebuilt. The other was decommissioned outright, rollback path deliberately surrendered. Both shutdowns required decisions the builds never asked me to make. Who is using this. What survives. Who gets more access than they normally hold, and who takes it back. Eighteen days after I explicitly preserved them, those artifacts were deleted by a cost-cleanup pass for the other project in the shared account.

Nothing in that sequence was an accident. Every step was correct at the moment it was taken.

What the evidence for shutting down actually looks like

The first thing a decommission needs is proof that nobody is using the system, and the first thing you learn is that your usage metric does not measure use. For the v1 backend of a small mobile product, the CloudWatch log group behind the API Lambda recorded 2,038 invocations over the final seven days of that window. Over the same window, the application's own request logger recorded 22 HTTP requests. The gap is a single EventBridge rule named ApiFunctionWarmKeep, firing on a five-minute schedule to keep the function warm: 288 invocations a day, and the daily invocation curve sat dead flat between 288 and 304 for the entire week. Roughly 99% of production traffic was the system talking to itself.

The warm-keep pings never reach the request logger, so no subtraction recovers the real figure. The decommission plan I wrote handled this properly: disable the warm-keep rule first, re-measure for fourteen days, and gate on zero real-user invocations for seven consecutive days before touching anything. Turning the synthetic traffic off is what makes the remaining number mean anything.

That phase never ran. The stack was deleted cold, on the strength of the seven-day snapshot alone, and the audit records that as a decision with an owner rather than an omission. I would make the same call again on a system with 22 requests a week. I would not make it without having done the arithmetic that showed the 22.

The harder finding was in those 22 requests: they came from one distinct user, and I could not work out who that person was. The v1 user_id is an internal Postgres UUID joined to the identity provider's subject only inside the database, the logs never emit an email and a user_id in the same line, and the RDS instance sat in private subnets with no bastion and no SSM session configured. The closest I could get was an eleven-second gap between a GET /users/me and one federated user's UserLastModifiedDate, which the audit document labels in its own words:

Strong correlation candidate (not proof):

A decommission is a forensics exercise you run against your own system with no attacker in it. If you cannot identify one user of your own production service on a good day with full account access, you have your answer about attribution on a bad one.

Freeze and destroy are different controls

The two shutdowns took opposite decisions about reversibility, and the question was not sentiment but who would need the data and when.

Situation	Choice	What you must own
Work may resume, data is reproducible	Freeze	The rebuild path, tested, with no dependency on the artifacts you kept
Work may resume, data is not reproducible	Freeze plus locked backup	The backup's protection mechanism, not its existence
Successor system is live and users are on it	Destroy	The named person who accepted the loss of rollback

Row three is the one people avoid. On the v1 backend the final snapshot was created late one evening and deleted about an hour later, retention set to zero days, on a recorded decision: the owner would not be rolling back to v1. Keeping a snapshot nobody will restore is not caution, it is an unowned copy of production data with a monthly bill attached. Row one is the one people get wrong in the other direction, by writing a resume procedure that depends on artifacts they have not protected.

Every grant you take, you have to enumerate back

Neither shutdown could be performed with the access that ran the system. The automation role for the frozen platform is assumable only by its GitHub Actions deployment identity, which was fine for applying infrastructure and useless for tearing it down from a laptop. The shutdown ran on borrowed privilege, and the discipline that matters is enumerating it. The destruction commit message lists the borrowing in full:

Temporary IAM grants used to perform the freeze have all been reverted:

• TerraformExecutionRole trust policy restored to github-actions-vertex-zt-only
• AmazonEC2FullAccess + CloudWatchLogsFullAccess detached from the role
• aaron-admin's temporary AmazonEC2FullAccess and vertex-zt-freeze-cleanup inline policy removed

Four grants, named in the artifact that also records the work they paid for. The revert is checkable by anyone reading git log, against a list written by the person who took the access rather than reconstructed later from CloudTrail.

The most interesting grant is the last one, and it exists because a control worked. The execution role carries a permissions boundary, and the boundary blocked terraform destroy from finishing the security group rule and VPC teardown. The same boundary had blocked the original build months earlier, with an error the progress notes captured verbatim: is not authorized to perform: ec2:CreateTags because no permissions boundary allows the ec2:CreateTags action. A ceiling narrow enough to stop your automation tagging a VPC at build time also stops it deleting one at teardown, and nobody reads a boundary as a teardown constraint until the teardown is what is running.

I did not widen the boundary. I attached AmazonEC2FullAccess and a purpose-built inline policy to my own user and finished the teardown by hand. The boundary held and the privilege moved to the human, the principal with the weakest change control in the account, doing the least reversible work of the project. The inline policy scoped its S3 statement to two named bucket ARNs and left the CloudWatch Logs, KMS, CloudFront and API Gateway statements on "Resource": "*". I scoped the part I was thinking about and wildcarded the rest, which is the ordinary shape of a policy written at the end of a long Sunday.

The strongest objection

The strongest objection is that this is ceremony priced far above the asset. The frozen platform cost 17 to 20 dollars a month; the decommissioned backend ran about 50, dominated by an idle NAT gateway rather than the database everyone assumes. A fourteen-day usage gate, a named owner for every preserved artifact and an immutable backup vault is more governance than either system was worth. Worse, a shutdown process that expensive is a shutdown process nobody starts, and systems left running on deprecated runtimes are a larger security problem than anything described here.

That is right about the ceremony, and I skipped the gate for exactly that reason. But the three things that made these two shutdowns checkable at all were free. The invocation arithmetic was one CloudWatch query. The resource inventory was one terraform state list saved to a file, which is how anyone can now see that 250 resources existed and what they were. The grant enumeration was four lines in a commit message. None of those is a program, a review board, or a quarter of work, and the absence of any one turns a shutdown into a guess.

A preserved artifact with no protection is a note

The pattern is preservation by README: an artifact is declared preserved in a document, and nothing in the platform enforces the declaration. It persists because the declaration is written at the moment of maximum care, by the person who most wants the artifact kept, and read afterward by nobody. NIST CSF 2.0 subcategory ID.AM-08 covers the life cycle of systems and data through disposal, and its implementation examples cover destroying and sanitizing data. The framework has no example for the thing you decided to keep.

It failed twice here, at two time scales. The freeze commit landed one morning with a "what was preserved" table listing the uploads bucket and the CloudTrail log bucket. Both were among the 250 resources in Terraform state. Fifty-five minutes later the destruction commit rewrote the table, because terraform destroy had taken them. The document said preserve, the state file said destroy, and the state file was the one with an executor.

The second failure took eighteen days. The freeze plan carried a safety rail in bold:

NEVER delete the Terraform state S3 bucket — it's the only record of what was deployed

That bucket and the final 20 GB database snapshot were deleted during a cost-cleanup pass on the shared account, the day after the unrelated project there finished its own decommission. The rail was real. It was also written somewhere that could only stop a person who happened to read it, in an account where the other project's cost work was legitimate and approved.

A snapshot taken through AWS Backup into a vault locked in compliance mode would have refused the deletion from every principal including root once its grace period elapsed, and AWS documents that grace period as a minimum of three days. This snapshot was not one: it came from a manual create-db-snapshot, so no vault covered it. The state bucket needed a different control again: S3 Object Lock, or a bucket policy the cost pass could not have satisfied. Two artifacts, two mechanisms, neither in place. I had eighteen days and I used none of them.

What would change my mind

I would drop the vault-lock recommendation if the next three shutdowns I run in a shared account leave their preserved artifacts intact for twelve months with nothing but a document protecting them. I doubt it, because the deletion here was not carelessness. It was a correctly motivated cost pass, executed by someone with legitimate access, against resources whose only defense was a sentence in a repository they had no reason to open. That is a threat model, not an accident.

The rail is still in the repo, and the file it protects is gone. What I changed is not the wording. The invariant it described now appears in the freeze document twice: once under the original freeze, struck through and annotated with the date it broke, and once under a post-freeze block that says what took it. The next person to read the resume instructions finds out from the document that the snapshot is gone, instead of finding out from AWS.