Salsburg, D. 2001.
``
The Lady Tasting Tea
-- How statistics revolutionized science
in the twentieth century''.
Owl Book.
Chapter 11. Hypothesis Testing
In their correspondence, Egon Pearson and Jerzy Neyman
explored several paradoxes that emerged from significance
testing, cases where unthinking use of a significance test
would reject a hypothesis that was obviously true. Fisher
never fell into those paradoxes, because it would have
been obvious to him that the significance tests were
being applied incorrectly. Neyman asked what criteria were
being used to decide when a significance test was applied
correctly. Gradually, between their letters, with visits
that Neyman made to England during the summers and
Pearson's visits to Poland, the basic ideas of hypothesis
testing emerged.
This led Neyman to two conclusions. One was that the power
of a test was a measure of how good the test was. The more
powerful of two tests was the better one to use. The
second conclusion was that the set of alternatives cannot
be too large. The analyst cannot say that the data come
from a normal distribution (the null hypothesis) or that
they come from any other possible distribution. That is
too wide a set of alternatives, and not test can be
powerful against all possible alternatives.