What is Evidence Strength?
We know Value Ratings are ideal for ranking complex options according to your personal preferences. However, in addition to the ranking, some situations call for knowing just how impressive the evidence is. This especially applies when decisions are critical, e.g., they carry a big impact, or involve highly sensitive matters.
Performance and Ability
It’s helpful to step back and recall our aim in measuring performance. We rely on performance to form judgments about ability, which is not directly visible to anyone. For example, a professional athlete possesses some unseen and remarkable ability; athletic performance is the only basis for us to infer what that level of ability is. Unfortunately, any measurement of performance is less than perfectly accurate. Performance results are a blend of signal and noise. To have greater confidence that measured performance reflects ability, we want more signal and less noise. Evidence Strength tells us this.
More Signal, Less Noise
The table below groups Value Ratings, which range from 0% to 100%, into eight tiers of increasing Evidence Strength.
Imagine coin-tossing scenarios. A Value Rating in the best tier is quite rare. Indeed, it is like flipping a fair coin nine times and getting heads all nine times. In contrast, a Value Rating in the vague tier would be like getting heads twice in two tries: maybe lucky, maybe not — but certainly a much more common occurrence. As you move up the tiers, it becomes harder to dismiss exceptional performance as chance or luck.
The eight Evidence Strength tiers are practical groupings. For example, a Value Ratings of 95% has greater Evidence Strength than a 94% Value Rating. But for our purposes, both can be approximated as being in the good tier — meaning a performance signal about as robust as tossing a coin four times and obtaining four successes.
For Critical Decisions, Pay Heed to Evidence Strength
Only a minority of Value Ratings occupy the more attractive Evidence Strength tiers, making them especially useful for teasing apart gradations of higher performance.
In the table above, about 64% the Value Ratings are associated with weak Evidence Strength. Roughly one-quarter land in the vague and emerging tiers. The remaining Value Ratings point to Evidence Strengths that range across five favorable tiers: good, strong, very strong, excellent, and best.
In short, if the choice is less than consequential, such as evaluating baseball teams or vacation destinations, the relative ranking of Value Ratings is usually sufficient. For weightier decision, attend to the Evidence Strength cues embedded in Value Ratings to identify reliably higher performance.