A “penalty for peeking” 2011/01/05Posted by Alex in : Uncategorized , trackback
(Nothing lewd follows.)
Just stumbled upon a fascinating short article by Rothman (1990) in the first volume of Epidemiology 1:43–6. The title of the paper says it all: no adjustments are needed for multiple comparisons. An exerpt from the end:
Suppose the drug C differs considerably in its effect from drug B. Will this difference be less worthy of attention when, sometime in the future, information on drug D comes along as part of the same research programme? Should an investigator estimate on the first day of data analysis how many contrasts ultimately will come along before making adjustments for multiple comparisons? Where do the boundaries of a specific study lie…?
What I’ve taken to doing when I have multiple nil hypothesis significance tests to perform is to write in the figure/table caption or methods the expected number of spurious positive findings conditional on the incredibly pessimistic premise that all nil hypotheses really are true (which is probably impossible for observational studies). Maybe I should cease even this?
This paper should be compulsory reading for everyone interested in statistics (and able to meet the prerequisites).