Environment Separation Is a Claim Until Someone Reads the Config

PROVIDERS_TABLE=aitc-providers-dev  DOCUMENTS_TABLE=aitc-documents-dev
MODELS_TABLE=aitc-models-dev        DOCUMENTS_BUCKET=aitc-documents-dev
WEBSITE_BUCKET=aitc-website-prd     ENVIRONMENT=prd

That is the environment block of aitc-static-generator-prd, the Lambda function that built the public site for a small service I ran tracking how AI providers change their terms of service. Four of the six values name development resources. The other two say prd, and one of those two is the label the service publishes about itself. The function fired on a rate(6 hours) schedule, returned HTTP 200 every time, wrote to the production bucket behind the production distribution, and served a site that loaded correctly in a browser.

Nothing in the account was wrong in a way that any check could see, because the checks all asked whether the function ran.

What production actually was in this account

The inventory that found this was not an audit. It was a list of things that had to be destroyed, and the list is the clearest evidence I have. The teardown script's DynamoDB step names five tables: aitc-documents-dev, aitc-models-dev, aitc-providers-dev, aitc-tflock-dev and aitc-tflock-prd. Exactly one of the five carries a -prd suffix, and it is a Terraform state lock table. The S3 step names six buckets, of which two are production: aitc-website-prd, which holds the published output, and aitc-tfstate-prd, which holds Terraform state.

So of eleven data-holding resources in the account, production owned one that contained anything a user would see, and it contained only the finished output. Every table and every source bucket underneath it was development.

Compute told the opposite story. The same script deletes eight Lambda functions, four EventBridge schedules and two Lambda execution roles, every one of them suffixed -dev or -prd in matched pairs, plus two API Gateway REST APIs it has to delete by ID. Separation was real at the layer where it cost nothing and absent at the layer that held the data. AWS's own Well-Architected guidance puts the boundary a level higher again: SEC01-BP01, "Separate workloads using accounts", rates the risk of not isolating environments at the account level as High. This was one account in us-west-2, and the entire boundary was a suffix in a resource name.

The boundary was removed on purpose, in one commit

It was not an accident, and that is the part worth sitting with. The commit that did it, from around the turn of the year, is titled "feat: Share data resources between dev and prod environments". Three of its seven bullets, in order but not adjacent:

• Dev creates and manages shared DynamoDB tables (aitc-providers, aitc-documents, aitc-models)
• ...
• Prod references these via data sources (read-only)
• Website bucket remains per-environment for UI testing isolation

The word doing the damage is "read-only". It is accurate about Terraform, where the production configuration references those tables through a data block rather than a resource block and therefore cannot modify them. It is not accurate about anything that runs. The production Lambda execution role, aitc-lambda-execution-prd, carries an inline policy granting dynamodb:PutItem, UpdateItem, DeleteItem and BatchWriteItem on arn:aws:dynamodb:us-west-2:*:table/aitc-* and on every index under it. Its development twin carries the identical policy. Two roles, one wildcard, no environment in it.

The commit also introduced the three lines that made the naming convention a lie. Everywhere else in that Terraform, resource names are built from env_suffix = "-${var.environment}". Here:

locals {
  providers_table_name = "${local.name_prefix}-providers-dev"
  documents_table_name = "${local.name_prefix}-documents-dev"
  models_table_name    = "${local.name_prefix}-models-dev"
}

A hardcoded -dev looks exactly like a rendered suffix in a console listing, a plan output, or a Lambda's environment variables. That is why it survived seven months.

Follow one generation through

The schedule fires. aitc-static-generator-prd scans aitc-documents-dev. For each item it builds the published shape, and the two lines that matter are (doc.versions || []).map(...) and (doc.deltas || []).map(...).

Now look at what the table was designed to hold. The schema comment in dynamodb.tf lists the documents table's attributes: source_url, current_version, current_checksum, current_s3_key, previous_version, previous_s3_key, status, last_checked, last_changed, and the two timestamps. There is no versions attribute and no deltas attribute. The table stored a pointer to the current document and a pointer to the one before it. The version history was produced by a pipeline that wrote JSON into the repository, and it never went into DynamoDB at all.

So doc.versions is undefined on every item, || [] turns that into an empty array, and the generator writes one entry per document, each carrying zero versions and zero deltas. It writes delta-index.json with total_deltas: 0. It writes meta.json stamped environment: "prd". Then it returns statusCode: 200 with a body reporting the document count it just processed, and logs the same count to CloudWatch. The inventory I wrote into the repository's own conventions file the day this was found put the live published index at 59 documents, zero versions and zero deltas; the account was torn down the next day, so that note is now the only place those figures exist.

The security property is lost at the ||. Everything downstream of it is a successful run of a correct program over the wrong data, and every count needed to catch it was computed, logged, returned, and read by nobody. The repository's own tracked website/data/*/versions.json files hold 66 documents, 633 versions and 549 deltas; cloudflare/seed.sql, generated from those files, contains 66 INSERT INTO documents, 633 INSERT INTO versions and 549 INSERT INTO deltas statements. Production was publishing an index emptier than the one in source control, and the difference between the two was the entire product.

The deploy step that turned this into an intermittent fault

The deploy script does something I still think was the right instinct, and it is the reason this went unnoticed for as long as it did. Its comment records the reasoning: data/ is excluded from --delete because the Lambda writes document content there "that does not exist in the repo", and removing it "would take out the archived document bodies". That was true and worth protecting. So the deploy runs two syncs: website/ with --delete --exclude "data/*", then website/data/ with neither. AWS documents the comparison rule for aws s3 sync plainly: a local file is uploaded if its size differs from the S3 object, if its modification time is newer, or if it does not exist in the bucket. The committed versions.json files are far larger than the empty ones the Lambda writes, so every deploy silently restored the real history. Then the six-hour schedule fired and flattened it again.

That is worse than a stable failure. A site that is always broken gets reported. A site that is correct for a few hours after each deploy and empty afterward produces one person seeing data, another seeing none, and no reproducible complaint. What I changed was not the deploy script. I held back the generator rewrite that would have shipped next, and its commit message says NOT YET DEPLOYED. Against the current data plane this Lambda emits an empty index -- see the following commit. Shipping a better generator against those tables would have published the empty index over the good committed one permanently, which would at least have been consistent.

The strongest objection

The strongest objection is that sharing one data plane was the correct engineering decision and I am blaming it for someone else's bug. Duplicating three DynamoDB tables and the whole document corpus behind them, so that a demo subdomain could have its own copy of data nobody was writing differently, is real money spent on symmetry. Plenty of teams deliberately point a lower environment at production data, or the reverse, and manage the risk with permissions rather than duplication. On that reading the sharing was fine, the empty output was the defect, and the fix is an assertion on what gets published rather than a second set of tables.

That is mostly right, and it is why the fix I would actually make is the output assertion. The part that does not survive is the label. ENVIRONMENT=prd was not inert configuration; the generator stamped it into meta.json on every run, so the service published a claim about itself that its own bindings contradicted. Share a data plane if the economics say so. Do not also emit a field asserting an isolation you have deliberately given up, because the next person to read that field will be reading it during an incident.

What I would write down instead

If the shared table is a decision rather than a mistake, the enforcement belongs in the one place that produces an error. The production role's grant covers the table and every index under it:

"Resource": [
  "arn:aws:dynamodb:us-west-2:*:table/aitc-*",
  "arn:aws:dynamodb:us-west-2:*:table/aitc-*/index/*"
]

Scoping both ARNs to the environment turns a silent misroute into a failed invocation:

"Resource": [
  "arn:aws:dynamodb:us-west-2:*:table/aitc-*-prd",
  "arn:aws:dynamodb:us-west-2:*:table/aitc-*-prd/index/*"
]

With that one suffix in both places, the first Scan against aitc-documents-dev returns AccessDenied, the function throws, the schedule records a failure, and the wrong binding is visible in the first six hours rather than after seven months. If the sharing is intentional, the exception is then written as an explicit second statement naming the shared table, which is a line a reviewer can see and argue with.

The assertion on output is the other half, and it is cheap because the numbers already exist. The generator counted the documents and the versions before it wrote a byte. A run that finds zero versions across every document, or a document count more than ten percent below the last meta.json, should refuse to write and raise. The owner is whoever owns the publishing job, and the trigger is the write itself, not a dashboard someone visits.

Go back to those six variables. Every one of them is a valid resource name, and nothing in that account ever put ENVIRONMENT=prd beside DOCUMENTS_TABLE=aitc-documents-dev and objected: not the plan, not the deploy, not the schedule, not the 200. A boundary that exists only in the gap between two strings nobody reads together is a naming convention, and the honest entry for it in a control column is "none". Write "none", then put the deny statement on the role and the assertion on the published output, where something other than a person has to read both.