How CI Security Gates Quietly Turn Themselves Off

The most dangerous state a security test can be in is not failure. It is not running at all, while the job that owns it reports success. A failing test produces a ticket, an argument, and a decision. A test that never executes produces a green check, and the green check is what everything downstream reads: the branch protection rule, the deploy gate, the release evidence, the slide.

Late last year I wrapped three test suites in describe.skipIf(...) in a zero-trust assessment platform I build. That made 38 tests conditional on an environment variable being present: 16 security tests on COGNITO_TEST_JWT_TOKEN, and 22 infrastructure tests on AWS_EXTERNAL_ID. Thirty-eight is my count of it( blocks in the three files at that commit, out of the 338 the same commit message reports passing locally. Every step that got me there was defensible. That is the part worth writing about.

Fifteen commits in two days, none of them wrong

The stabilization run for that pipeline's new workflows spans two days: fifteen commits after the workflow files themselves landed, all on develop. Times below are offsets from the first of them.

Offset	Change	Effect on the gate
T+0	--legacy-peer-deps on npm ci in all four workflows	Install completes; the React 19 peer conflicts stay unresolved
T+10m	if-no-files-found: error on artifact uploads	An empty build output now fails the job loudly
T+21m	include-hidden-files: true	Real cause found: .open-next is a hidden directory
T+1h19m	eslint-config-next moved back two major versions	Lint runs, against a config two majors behind
T+3h19m	continue-on-error: true on five admin-portal steps in ci.yml	Install, lint and type-check can fail green
T+3h23m	Same in deploy-dev.yml, plus artifact upload back to if-no-files-found: warn	The T+10m tightening reverted for one artifact
T+33h	CI generates a Cognito JWT for the security job	Security tests receive real credentials
T+33h20m	describe.skipIf(...) on three test files	38 tests skip when their credential is missing

Look at rows two and six. I made an empty artifact a hard failure because I could not see why the upload was producing nothing, and found the actual cause eleven minutes later. The tightening had already done its job. Three hours on, I put the admin-portal artifact back to warn anyway, so an unrelated build problem would stop blocking deploys. warn is the documented default for that input in actions/upload-artifact: it prints a warning and does not fail the action. The loud version of the control lived for one working morning.

The three ways a gate goes quiet

Three mechanisms turned that gate off across those two days, and they go quiet in different ways. The first is the conditional skip, and it is the only one that leaves the control's code untouched. describe.skipIf(skipInfraTests) still reads as a test suite; the suite simply resolves to zero executed assertions when CI === 'true' and AWS_EXTERNAL_ID is unset. Nothing in the file says "this ran" or "this did not."

The second is continue-on-error, which is more honest than it looks. GitHub's own contexts documentation distinguishes steps.<step_id>.outcome, "the result of a completed step before continue-on-error is applied," from steps.<step_id>.conclusion, "the result of a completed step after continue-on-error is applied," and states plainly that "when a continue-on-error step fails, the outcome is failure, but the final conclusion is success." The truth exists in the data model. It is conclusion that renders as the check, and conclusion that a required-status-check rule evaluates.

The third is the downgrade. Moving eslint-config-next from 16.x to 14.x kept lint running, and lint running is better than lint erroring out on a circular-structure crash. It also means the security-relevant rules being enforced are whichever ones shipped two major versions ago. This is the quietest of the three, because a passing lint step gives no signal at all about which rule set produced the pass.

The skip fires on a four-character string

The change that interests me most is inside the security test helper. Before, an absent token was an error:

if (!token || token === '') {
  throw new Error(
    'COGNITO_TEST_JWT_TOKEN not found in environment. ' +
      'Run with: doppler run --config dev -- npm run test:security'
  )
}

After, it is a value:

if (!token || token === '') {
  // Return empty string - tests will be skipped if no token available
  return ''
}

The suite is then guarded by describe.skipIf(!hasJwtToken()), and hasJwtToken() returns true only when the token is present and longer than 100 characters. Follow one CI run through that. The security job runs aws cognito-idp initiate-auth --auth-flow USER_PASSWORD_AUTH ... --output text --query "AuthenticationResult.IdToken" and assigns the output to TOKEN. The ellipsis is where the client ID and the test account's credentials sit.

Cognito's API reference says AuthenticationResult "is only returned if the user doesn't need to pass another challenge" and that when a challenge is required the service returns ChallengeName, ChallengeParameters and Session instead, with an HTTP 200. The CLI exits zero. The JMESPath query resolves to null, and --output text renders a null result as the literal string None. TOKEN becomes None. Four characters is not more than 100, so all 16 security tests skip, and the job goes green.

The security property is lost at exactly one hop: the --query expression, which returns the same shape for "there is no token" and "there is no token yet, because this account needs to set up MFA first."

Green, red, and the state that is missing

The test runner is not hiding anything. Vitest's default reporter prints a summary line in the documented form Tests 10 passed | 3 skipped (65), so the skip count is sitting in the run log of every job. The problem is that nobody opens the log of a green job. The check status carries one bit, and a skipped test contributes to neither side of it.

So the pipeline has two states where it needs three. Pass means the control ran and the system satisfied it. Fail means the control ran and the system did not. Neither of those is what happened when 38 tests were never scheduled, and the third state has no representation anywhere a person or a policy will encounter it.

Epic deferral. The same absence shows up one layer out, in what a suppression records about its own future. A suppression carries a pointer to that work, the pointer is a code comment, and the comment is the only place the work exists. The string tracked for Epic 5 appears 11 times across ci.yml and deploy-dev.yml in this repository, nine of them attached to a continue-on-error: true.

Neither docs/project/epic-5-security-hardening.md nor docs/project/epic-5-prod-v1-readiness.md mentions ESLint or continue-on-error at all, and both mention the admin portal only as a feature area, never its build. No open or closed issue in the repository names any of it either. The pattern persists because writing the comment is the moment of maximum sincerity and reading it is nobody's job. The cheapest thing that breaks it is giving the suppression an expiry the pipeline itself can read, so the workflow fails on the date rather than the intention.

The strongest objection to alerting on skips

The strongest objection is that skip conditions are load-bearing, and that alerting on them manufactures a second stream of noise you will mute within a fortnight. A test that authenticates against a live Cognito user pool genuinely cannot run on a pull request from a fork, because forks do not receive secrets. describe.skipIf is the correct primitive for that, not a workaround for it. Page an engineer every time it fires and you will page them on every external contribution, they will write a filter, and the filter will swallow the case you actually cared about along with the rest.

That is right, and I do not have a clean general way to express the difference between an expected skip and an unexpected one. What survives the objection is the asymmetric version: do not alert on skips, alert on skips in the one context where the credential was supposed to be present, which for this pipeline is a push to develop and is a single boolean in a workflow condition. I should also concede that I built the reporting once and did not generalize it, because the deployment notification job in deploy-dev.yml posts a commit comment reading :warning: Admin Portal deployment skipped (ESLint version conflicts - tracked for Epic 5) whenever that job does not succeed. One of ten continue-on-error uses across the two workflows reports itself. The mechanism was never the hard part.

What to check in your own pipeline

If you own a pipeline with security or infrastructure tests in it, this is an hour:

1. Grep every workflow file for continue-on-error, || true, allow_failure and soft_fail, and count the hits.
2. Of those, count how many emit a message somewhere a human reads without opening the job log.
3. Grep the test tree for skipIf, .skip(, xit(, pytest.mark.skipif and t.Skip(.
4. For each hit, write down the environment variable it reads and which job is supposed to set it.
5. Compare the test count in your last green CI run against a full local run.
6. Open the ticket referenced by every suppression comment and confirm it exists.

Step five is the one that tells you whether you have a problem: if the two numbers differ and nobody can say by how much or since when, your green is not measuring what you think it is. Step six is the one that tells you how long you have had it.

Weeks later, still counting

The change that added describe.skipIf is still the most recent commit on develop. Nothing has landed on that branch since, which puts those 38 conditional tests several weeks old with no follow-up, and leaves no way to tell from the outside how many runs in between executed them.

The rule I take from this: any control that can decide not to run must emit that decision on the same channel it would have emitted a failure on, because a channel nobody watches when things are fine is not a channel. The boundary is that this needs somewhere for a third state to live. A commit comment, a metric, a required job whose only work is to assert the expected test count. If the only output your pipeline produces is a red mark or a green one, you have two states and you will keep losing the one that matters.