Interpreting P values

Suppose you have just finished the measurements in a simple experiment to compare the abundance of weeds on 1-m2 plots sprayed either with a weedkiller in solution or with the same volume of water alone. You adopt the null hypothesis: that the weedkiller has no effect, and make a suitable test of this hypothesis (t-test or U-test, as appropriate) obtaining the result P = 0.04. That is to say that, if the hypothesis is true, then in a large number of similar trials one would expect to have got mean values at least as different as these were in about 4% of trials. The statistics stops at this point. So what next?

Anything further is a judgement by you and your readers. A convention has grown that for routine work of no special importance, if P ≤ 0.05 then, as practical people we think we will make fastest progress by concluding provisionally that the hypothesis is false while accepting that on average 1 in 20 of these provisional conclusions will be incorrect. But the value 0.05 is arbitrary.

• If we had only four digits on each hand it seems likely that the value would be 3 / 82 (= 0.047 in base 10) rather than 5 / 102 = 0.050.

• If the experiments are important and cannot be repeated for some reason (expense, no more material, unavailable equipment) then you may argue that you are going to reject the hypothesis provisionally with P as high as 0.1 (for example).

• If your life depends on avoiding wrongly rejecting the hypothesis then you might be inclined to require that P be ≪ 0.000 000 1.

• And if your life depends on avoiding wrongly rejecting the hypothesis yet there is a reward of 1 000 000 €/£/$ for correctly rejecting it, then you might be inclined to gamble at P < 0.000 1 or even P < 0.001. This is the sort of subconscious judgement you make every time you get into a car: the weighing of advantages and risks of an adverse outcome.