鈥淥ver 80 per cent of women say that this shampoo leaves their hair healthier and shinier.鈥� Such claims are common in advertising for all manner of consumer products. What they might not tell you is that only five women tested the shampoo. And of the four who certified its miraculous effect, one or two probably ended up with nicer hair purely by chance, or simply imagined the results.
Similar caveats apply to the effectiveness of medical treatments. Curing six out of 10 patients is promising. Curing 300 out of 500 is the same success rate, but far more convincing. 鈥淭he sample size in a test is absolutely crucial in deciding whether any apparent improvement could have happened by chance alone,鈥� says Spiegelhalter.
The standard procedure for such trials is the one established by Bradford Hill over 60 years ago: new medical treatments are tested in randomised controlled trials (RCTs), in which volunteers are randomly allocated to a study group that receives the new treatment or a control group that receives a placebo or existing treatment. 鈥淵ou can think of an RCT almost as a measuring instrument to measure a treatment鈥檚 effectiveness,鈥� says of the UK Medical Research Council Biostatistics Unit in Cambridge. To make sure any instrument is sensitive enough for its job, you need to assess how big an effect it is expected to measure.
Advertisement
Working out the size of the expected effect requires an analysis of past studies or the results of tests on animals. In the case of an RCT, the smaller the expected effect, the more people you need to enrol in your trial, and vice versa.
Another important consideration is the level of significance the trial is expected to achieve 鈥� that is, the likelihood that a useless treatment will register the effect you are after as a result of chance alone. RCTs are usually designed to achieve a 5 per cent significance level. This means that even if the drug is useless, it will register a positive result by chance in 1 out of 20 trials. For that reason, says Spiegelhalter, drug licensing authorities do not usually consider a single study sufficient evidence to approve a new drug. Repeat trials are needed.
So next time you hear of public acclaim for a miracle cure or wonder shampoo, ask three questions. How many people was it tested on? Was it tested in an RCT? And was the result confirmed by a second, independent test?
Read more in our web special: 鈥�Spin doctors: The truth behind health scare headlines鈥�
