Decomposing a judgment into a matrix multiplies subjective calls instead of eliminating them

Breaking a hard judgment into rows and columns looks like discipline. In practice, every cell of the grid is its own unstructured micro-judgment, so the structure creates more occasions for analysts to diverge, not fewer. The intuition the method was built to drive out does not leave; it relocates into the dozens of small choices about how to fill in the structure.

The case is Analysis of Competing Hypotheses, the CIA's flagship technique: list evidence in rows, hypotheses in columns, rate each cell for consistency. Chang, Mandel, and Tetlock pointed out that consistency is never defined, so the engine of the method runs on a word each analyst reads differently:

Reading of "consistent"	What the analyst actually computes
Forward probability	how likely the evidence is if the hypothesis is true
Inverse probability	how likely the hypothesis is given the evidence
Plausibility	whether the two can coexist without contradiction
Resemblance	whether the evidence looks like the hypothesis

Mandel called the result a covert greenhouse for noise. In one experiment, analysts trained in ACH did worse on a probabilistic hypothesis-testing task than colleagues left to their own reasoning. The technique spread anyway, mandated by US law in 2004, because endorsement by prestigious insiders substituted for testing.

The pattern generalizes to any scoring rubric or weighted framework: structure standardizes the paperwork, not the judgment. Related: [[Self-sealing hypotheses cannot be broken with more evidence]].

Source claim: Decomposing a judgment into structured parts relocates intuition into many undefined micro-judgments, amplifying noise rather than removing bias.