Which do you prefer: chocolate-covered broccoli or chocolate-covered raw chicken?
Comparative readability testing works about like this question. It can tell us people strongly prefer A over B. But whether we’re talking delicious or disgusting it can’t say.
In fact, most testing actually prevents us from finding that out. Readability is based on two variables: how difficult the document and how motivated the reader. When you know you’re in a test, your motivations change — and with them, any chance to measure.
Used properly, testing is a great tool. But it’s good to remember it’s not the only tool, nor always the best. In simplification, as in cooking, you still need expertise. Sure, with enough testing, you might get from chocolate-covered raw chicken to pollo en mole poblano. But we already know that tastes good. And a good chef already knows how to make it.