What Automating a Zero Trust Assessment Taught Me About Maturity Scores

I have put maturity scores into client reports for years without ever having to state exactly how one was computed. Late last year I had to. The assessment platform I was building turns 33 questions into eight bucket scores on a 1-to-4 scale, and a scoring function will not accept judgment as an input. I am writing this five weeks later, long enough to see which of those decisions I made and which the code made for me.

One commit that month touched four files, 399 insertions and 155 deletions, under the message "fix(story-3.2): Bug fixes for answer persistence and scoring". Three of those four changes altered what number a client would see for the same set of answers. None of them touched a question, a weight, or anything about the organization being measured.

That is the subject of this post. A maturity score that moves when the organization did not is not a finding. It is a defect in the instrument, and the instrument is usually the part nobody audits.

I expected the fights to be about content

The fights I planned for were about content. Which questions belong in the Identity pillar, whether "we have single sign-on for most applications" is an Initial or an Advanced answer, how to phrase a question so that a network engineer and a compliance manager read it the same way. Those are real arguments and I had them.

I expected the arithmetic to be trivial, because it is. Sum the weights of the selected options, divide by how many there are, look the result up in a band table. The whole calculation in scoring.ts fits in about twenty lines.

What I did not expect was that every hard decision would be about absence. Not what an answer is worth, but what a non-answer is worth, and how many kinds of non-answer a system can produce without telling anyone.

Lesson 1: zero means unmeasured and 25 means worst

A pillar with no answered questions scores zero. The line is not ambiguous:

pillarScores[pillar] = count > 0 ? roundScore(sum / count) : 0

Zero is not on the scale, which runs 1 to 4. The platform stores scores as percentages using Math.round((overallScore / 4) * 100), so the worst maturity a measured pillar can report is 25. A pillar reporting 0 has not been measured at all. Those two numbers sit next to each other on the same report page in the same units, one meaning "traditional, no controls" and the other meaning "I have no idea".

It gets worse downstream. Recommendations and gaps are both generated from the same filtered list, filter(([, score]) => score > 0 && score < 4). A pillar scoring zero is excluded from both, so the pillar the client answered nothing about produces no recommendation, no identified gap, and reads as clean.

The schema has the field that would have caught this: confidence_score, created in db-init.ts as "confidence_score" INTEGER, which the Prisma model annotates // 0-100 based on completion. Somebody designed the right field. In scoring.ts, the value returned to the report page is this:

confidenceScore: 0,

Hard-coded. The field that exists to say how much of the assessment was answered returns a constant that reads as "no confidence" and means "never computed". The cheapest fix is not a better rubric. It is refusing to render any pillar score without the response count that produced it.

Lesson 2: a stored answer can silently stop counting

The scoring query joins each response to the option the respondent selected, and the option carries the weight. Before that commit, the match ran through a correlated subquery:

-- before
JOIN question_options qo ON q.id = qo.question_id
WHERE qr.assessment_id = ${input.assessmentId}
  AND qo.id = (
    SELECT jsonb_array_elements_text(qr.response_value->'selectedOptions')::text LIMIT 1
  )::uuid

If that subquery yields nothing, qo.id = NULL is never true and the response leaves the result set. It does not error. It does not log. The row is simply not there when the average is taken. The same commit changed how answers were written, adding a ::jsonb cast in questions.ts where the code had been passing a stringified object, which is why the fixed query has to tolerate more than one stored shape:

-- after
CROSS JOIN LATERAL (
  SELECT
    CASE
      -- Array format: extract first element
      WHEN jsonb_typeof(qr.response_value->'selectedOptions') = 'array'
           AND jsonb_array_length(qr.response_value->'selectedOptions') > 0
      THEN (qr.response_value->'selectedOptions'->>0)
      -- String format: use directly
      WHEN jsonb_typeof(qr.response_value->'selectedOptions') = 'string'
      THEN (qr.response_value->>'selectedOptions')
      ELSE NULL
    END as option_id
) selected_opt
JOIN question_options qo ON qo.id = selected_opt.option_id::uuid
WHERE qr.assessment_id = ${input.assessmentId}
  AND q.id = qo.question_id
  AND selected_opt.option_id IS NOT NULL

AND selected_opt.option_id IS NOT NULL on the last line is the fix. An unparseable answer is now an explicit exclusion rather than a row that quietly fails to match, and an exclusion is something a count can find.

Now follow what a dropped row did to the number before that line existed. The overall score is roundScore(totalWeight / responses.length), and responses.length counts the rows that survived the join, not the questions the client answered. A dropped response leaves the numerator and the denominator together, so the score does not fall toward zero. It drifts toward the average of whatever survived. Drop the low answers and maturity goes up.

The only place this was visible was completionPercentage, computed from the same survivor count, and the status flip to COMPLETE, gated on responses.length >= totalQuestions. So the defect reached me as a workflow bug, filed under the string the user sees: "Failed to complete assessment. Please try again." Nobody reported a wrong score, because nobody could have. And recalculation updates the existing analysis_results row rather than inserting one, so the previous number is gone the moment the new one is written.

Lesson 3: one answer in thirty-three moves the maturity band

Here is that sensitivity on a case I can show you. The golden dataset shipped with the platform, data/golden-dataset.json version 2.1.0, contains a scenario named scenario-improving: 33 responses, 17 answered at weight 3 and 16 at weight 2. The sum is 83, the mean is 2.5152, and the platform reports 2.52.

The band boundary in getMaturityLevel is 2.5. Below it the organization is Initial; at or above it, Advanced. Change one answer from a 3 to a 2 and the sum is 82, the mean is 2.4848, and the report says Initial. One question out of 33 is worth 0.0303 on the mean, and this organization sits 0.0152 above a band edge. The band an executive repeats to a board is decided by a single respondent's reading of a single question.

The weighting underneath is not a decision anybody made either. The overall score averages responses, not pillars, so each bucket's influence is whatever share of the questions it happens to hold. In that dataset each of the five core pillars carries five questions, the two cross-cutting ones carry three each, and Governance carries two. Governance is 6 percent of the score. That number is an artifact of how many questions got written, not of how much governance matters.

The framing is off in a second way. CISA's Zero Trust Maturity Model v2.0, published April 2023, defines five pillars plus three cross-cutting capabilities. The scoring engine has eight equal buckets. The project README describes "maturity level calculation across five CISA pillars" while the report page tells the reader its number is "based on N responses across 8 Zero Trust pillars". Both strings shipped.

The fix for all three is a corpus that rots

All three lessons have the same fix: a corpus of known answers with known expected scores, run on every release. That is what data/golden-dataset.json is, versioned properly, carrying schemaVersion: 2.1.0 and compatibleWith: ["3.1.x"] so it declares which builds it is valid against.

I have already watched one of these rot. An investigation note I wrote a couple of months earlier records a seeding run that produced 63 question responses where roughly 580 were expected. The dataset had been authored against a question library using codes like MFA-001 and SIEM-002; the deployed library, 60 questions at that point and not yet regenerated to the 33 this post opens with, used DEV-001 through DEV-008 and VIS-001 through VIS-006. Eight codes matched out of 58. The seeding job reported success and wrote 10 assessments and 10 analysis results, each scored on six to eight answers.

And the full validation suite does not run. In e2e/tests/mvp-validation/golden-dataset.spec.ts all three suites are test.describe.skip, with a comment explaining why: "Long-running validation test (20+ min)". The regression test for the measurement is the one test the pipeline skips, because it is the one test that has to answer 33 questions through a browser seven times. I do not have a good answer to that yet. Running the scoring function directly against stored responses would be fast and would miss exactly the defect in lesson 2, which lives between the browser and the column.

Automation did not create these decisions

The strongest objection is that I have described bugs in one immature application and dressed them up as an indictment of maturity scoring. A consultant with a spreadsheet has no LATERAL join to get wrong, no JSONB column to double-encode, and no auto-recalculation loop. Building the instrument in software created most of these defects; not building it would have avoided them. On that reading the lesson is about my code, and it does not transfer.

Half of that is right. The specific defects are artifacts of this stack and you will not find them in a workbook. But the four decisions underneath them are not optional, and a spreadsheet does not remove them: what an unanswered question is worth, what an answer nobody can parse is worth, how much weight a pillar carries, and where the band edges fall. Automation did not create those decisions. It made them visible, gave them line numbers, and put them in a diff where somebody could argue with them.

Here is the part I concede, and it is larger than the objection. Everything in this post is about reproducibility, and reproducibility is not validity. I can now show that the same answers produce the same score. I have no evidence at all that those 33 questions measure zero trust maturity, and a perfectly stable instrument can be stable about nothing. Fixing the arithmetic bought me the right to ask the harder question, not an answer to it.

What to ask anyone selling you a maturity number

The transferable move is a question you can put to anyone who has sold you a maturity number, including me. Ask them to re-score last quarter's answers under this quarter's rubric and show you both numbers. If the two differ, the difference is theirs to explain before any trend line gets drawn. If they cannot produce last quarter's answers at all, the trend is decoration and you should stop briefing it.

What that question does not test is the questions themselves. A rubric can be reproducible in every way described here and still be asking the wrong 33 things. The commit that started this post is filed under answer persistence, and that is honestly what I set out to fix. What it changed was which answers counted, on every assessment the platform had already scored, and the old numbers were overwritten as it went. No client was ever going to file that ticket.