A WELCOME CRITIQUE OF THE IMPROPER USE OF STATISTICAL SIGNIFICANCE IN SCIENCE

The critique below is based on Bayesian statistics, which are fine in theory but difficult in application. Nonetheless, there is little to question in what they say below about the excessive credence given to statistical significance in medical research -- or in psychological research, for that matter.

I myself favour a much simpler approach but one that will endear me to very few epidemiologists. I favour a survey-wise rather than a contrast-wise error-rate approach. That sounds more profound than it is. I am simply saying that we should have much higher standards for significance when an observed relationship is just one of many fished out of a large matrix of correlations: "Data dredging", as it is called. Most of what is reported in the epidemiological literature would fall to non-significance if that approach were applied.

Let me give a rough example: A survey that obtained data on 100 variables would not be terribly unusual. But such a survey would give rise to nearly 5000 (100 x 100 /2) unique relationships (correlations). But the conventional level of probablity for accepting significance is 5%. So on chance alone nearly 250 (5%) of those 5000 correlations would rate as "significant". And lots of such correlations have been reported as "significant" in the epidemiological literature. Random associations have been reported as if they tell use something!

Sadly, that malpractice is most unlikely to change. Given the relatively few large databases available, it is part and parcel of epidemiology, in fact. Meta-analyses (looking at a whole lot of different research reports together) have some potential to overcome the problem but I won't depress people any further by going into the large problems of meta-analyses. I can point to some shocking ones!

I myself was rather subject to similar temptations in my published research. I often administered large surveys and based quite a few articles on correlations observed in that single survey. The sort of relationships that I routinely dismissed as too weak to enable any positive inferences would however have been greeted with something approaching a Nobel prize had they been observed in epidemiological research. I used to accuse my fellow psychologists of making mountains out of molehills but in medical research they routinely make mountains out of pimples!

Effect of Formal Statistical Significance on the Credibility of Observational Associations

John P. A. Ioannidis

The author evaluated the implications of nominal statistical significance for changing the credibility of null versus alternative hypotheses across a large number of observational associations for which formal statistical significance (p < 0.05) was claimed. Calculation of the Bayes factor (B) under different assumptions was performed on 272 observational associations published in 2004–2005 and a data set of 50 meta-analyses on gene-disease associations (752 studies) for which statistically significant associations had been claimed (p < 0.05). Depending on the formulation of the prior, statistically significant results offered less than strong support to the credibility (B > 0.10) for 54–77% of the 272 epidemiologic associations for diverse risk factors and 44–70% of the 50 associations from genetic meta-analyses. Sometimes nominally statistically significant results even decreased the credibility of the probed association in comparison with what was thought before the study was conducted. Five of six meta-analyses with less than substantial support (B > 0.032) lost their nominal statistical significance in a subsequent (more recent) meta-analysis, while this did not occur in any of seven meta-analyses with decisive support (B < 0.01). In these large data sets of observational associations, formal statistical significance alone failed to increase much the credibility of many postulated associations. Bayes factors may be used routinely to interpret "significant" associations.

American Journal of Epidemiology 2008 168(4):374-383

Childhood obesity a myth, say Australian food advertisers

How pesky of them to look at the evidence for popular claims!

The advertising industry has denied there is any link between food advertising and childhood obesity. At a federal Parliamentary inquiry into obesity in Australia held in Brisbane today, MPs were also told that advertising standards prohibited food being advertised as healthy in Australia. Australian Association of National Advertisers executive director Collin Segelov claimed CSIRO research, yet to be released, would show no significant increase in childhood obesity since the last study in 1995. "I'm not only arguing that advertising is not the cause of a childhood obesity epidemic, but that there is no epidemic," Mr Segelov said.

"The incidence of obesity amongst schoolchildren in Australia has shown no significant increase since 1995 [The findings in the USA are similar]. "This makes the notion of an obesity epidemic, as continually put forward by academic activists and others - quite irresponsibly in my opinion - quite misleading, if not an utter nonsense." Mr Segelov said food advertisers remained committed to a broader, more holistic approach to obesity.

Foundation for Advertising Research founder Glen Wiggs said an Australian food standard specifically forbade the use of the word "healthy" in food product advertising. Professor Wiggs said the repeal of the standard had been delayed again and again, but was now scheduled for April next year. He told AAP that research from England indicated advertising only affected food choices by children in a modest way, and their parents held far more sway. Advertising was an easy and cheap research target for authorities, but very little research was undertaken into how the home affected food choices, he said. "Children tend to imitate their parents," Prof Wiggs said.

Mr Segelov told the committee that once the food standard that currently prohibits the labelling of a Tasmanian apple as healthy was dropped, it would be imperative for advertisers to promote healthy products. He said television advertising bans would not work because advertisers would merely switch platforms, and children were already moving from television to other forms of "screen time" - the internet and pay TV.

Comment is being sought from the federal Department of Health and Ageing on the latest nutrition survey.

Source