Statistical significance

Below is the first part of an article in the NYT that looks at why we use tests of statistical significance in scientific research. The issue is a relevant one for medical research as every medical journal article one looks at does use and report tests of statistical significance for the findings described in it.

The discussion below is quite right in reporting that many statisticians think we are too lenient in what we accept as statistically significant. I think that too. Accepting a 5% chance that a correlation may be due to chance alone seems risky in medical research. Medical research should surely observe high standards of proof before conclusions are drawn.

What the article overlooks, however, is that statistically aware researchers have never been much impressed by the 5% standard. It is quite common for results significant by that standard to be seen as preliminary or tentative. For researchers to report their results with any confidence about the reality of what they report, a 1% standard has long been the informal criterion. And to the extent to which that is so, the criticisms below rather fall by the wayside: There is already a stronger standard in informal use.

I taught statistical analysis in a major Australian university for a number of years so have had some opportunity to reflect on why we use tests of statistical significance. And I think the major point that is rather overlooked below is that an effect can be significant in a statistical sense but not in any other sense. All a significance test does is exclude randomness. But even a tiny correlation can be shown as non-random if the sample size is large enough. But a tiny correlation may be of no practical importance or use at all.

So the function of significance testing is simply to act as a filter. Such a test enables us to say of some correlation: "This correlation is NOT EVEN statistically significant". And in that case its significance in any other sense is unlikely to be worth bothering about. It is, in other words, just a very preliminary filter which we use to help sort out which correlations may be worthy of our attention

In recent weeks, editors at a respected psychology journal have been taking heat from fellow scientists for deciding to accept a research report that claims to show the existence of extrasensory perception.

The report, to be published this year in The Journal of Personality and Social Psychology, is not likely to change many minds. And the scientific critiques of the research methods and data analysis of its author, Daryl J. Bem (and the peer reviewers who urged that his paper be accepted), are not winning over many hearts.

Yet the episode has inflamed one of the longest-running debates in science. For decades, some statisticians have argued that the standard technique used to analyze data in much of social science and medicine overstates many study findings — often by a lot. As a result, these experts say, the literature is littered with positive findings that do not pan out: “effective” therapies that are no better than a placebo; slight biases that do not affect behavior; brain-imaging correlations that are meaningless.

By incorporating statistical techniques that are now widely used in other sciences — genetics, economic modeling, even wildlife monitoring — social scientists can correct for such problems, saving themselves (and, ahem, science reporters) time, effort and embarrassment.

“I was delighted that this ESP paper was accepted in a mainstream science journal, because it brought this whole subject up again,” said James Berger, a statistician at Duke University. “I was on a mini-crusade about this 20 years ago and realized that I could devote my entire life to it and never make a dent in the problem.”

The statistical approach that has dominated the social sciences for almost a century is called significance testing. The idea is straightforward. A finding from any well-designed study — say, a correlation between a personality trait and the risk of depression — is considered “significant” if its probability of occurring by chance is less than 5 percent.

This arbitrary cutoff makes sense when the effect being studied is a large one — for example, when measuring the so-called Stroop effect. This effect predicts that naming the color of a word is faster and more accurate when the word and color match (“red” in red letters) than when they do not (“red” in blue letters), and is very strong in almost everyone.

“But if the true effect of what you are measuring is small,” said Andrew Gelman, a professor of statistics and political science at Columbia University, “then by necessity anything you discover is going to be an overestimate” of that effect.

More here




Can it really be true that statins won't stop heart attacks?

Cochrane reviews aim to be the most thorough and comprehensive in the medical literature so are generally regarded as highly authoritative

Confused about statins? Hardly surprising, when even the experts seem to disagree. Last week, a major report suggested that if you hadn’t had a heart attack or a stroke, ­taking one of the ­cholesterol-­lowering drugs was probably a waste of time. That’s because the chance of them preventing a heart attack was very small.

But other experts rejected the report by the respected Cochrane Library, saying it hadn’t included the latest studies. ‘The quality of the data showing the effectiveness and safety of statins is remarkably high,’ argued Dr Colin Baigent of the Cholesterol Treatment Trialists’ Collaboration in Oxford. ‘We now have a very large database of patients that show clear benefits.’

There are just over two and a half million people living with heart disease in the UK and there’s no doubt many will be ­benefiting from statins.

However, seven million Britons take them and the number is rising. While some of these people are at high risk of heart disease and may be helped by the drugs, several ­million others are taking them when the evidence for their benefits is conflicting.

And now to add to the confusion, an ­influential cardiology organisation known as The Joint British Societies is expected to announce even more people should be put on the drugs and given them earlier.

So who is right, why has it taken so long for these doubts to appear, and what else could you be doing? Good Health asks the experts.

I’ve never had a heart attack, do I need statins?

That depends on your age and how healthy you are. The latest report found little evidence that taking a statin would protect people from having a first heart attack unless their risk was high. This risk is calculated by your doctor according to a number of factors including your cholesterol levels.

But if you are female or over 65 and at low risk, the Cochrane review says it’s even less clear. That’s because most ­trials involve white, middle-aged males so the results don’t necessarily apply to anyone else.

But some experts say that I should take them . . .

These experts were probably ­relying on studies that some people now say were flawed. This is because they included patients who already had heart problems — there’s little doubt that statins help these patients, so their inclusion skews the results. Critics say you can’t use this evidence to justify treating healthy people with statins.

The evidence for statins can also be made to seem more favourable that it really is. One technique used by drug companies is ‘simply to not say very much about negative findings’, says Dr Shah Ebrahim, senior author of the latest review.

The Cochrane review, carried out by researchers from the London School of Hygiene & Tropical Medicine and the University of Bristol, closely analysed 14 controlled trials that involved 34,272 primary prevention patients — that is, people who’d never had a heart attack.

It found ‘only limited evidence that primary prevention with ­statins may be cost-effective and improve patient quality of life’. The small size of the benefit is vividly illustrated by this fact: out of 1,000 primary patients taking a statin, only one death from heart disease would be avoided.

If benefits are that small then it becomes more important to balance them against the side-effects.

So what are the side-effects?

On the positive side, the review didn’t find any evidence that the drugs cause some of the adverse reactions that people have worried about in the past, such as cancer, low mood or anger or increased deaths from violence or suicide.

But that still leaves quite a number that you have to weigh against a small benefit. Statins are well known to cause muscle problems, including muscle pain, fatigue and weakness. Estimates of how common they are vary widely — from 1 per cent to 20 per cent.

Other potential ­reactions include cataracts, acute kidney failure, and moderate or severe liver dysfunction, said to be rare. Recently the Medicines and Healthcare products Regulatory Agency warned about some ­additional risks — sleep ­disturbances, memory loss, sexual dysfunction, depression, and (very rarely) interstitial lung disease.

However the review points out that the trials don’t give nearly enough information about side-effects. Over half the trials didn’t report on adverse effects, and there has been no attempt to assess the risk of some potentially serious side-effects such as ­cognitive impairment or the risk of diabetes when cholesterol is ­lowered too much.

What if I’m only ‘at risk’ of a heart attack?

The Nice guidelines say that if you have some of the familiar risk factors for heart disease — are male and over 55 (65 for a woman), have high cholesterol, smoke or are overweight — which gives you a 20 per cent or more chance of a heart attack in the next ten years, you should be prescribed statins.

The Cochrane report doesn’t change that advice. It says ‘it is likely the benefits of statins with a raised risk of heart disease are greater than potential short-term harms’, but warns: ‘Long-term effects (over decades) remain unknown’.

Dr David Tovey, head of the Cochrane Editorial Unit, says: ‘This report is a warning against expanding statin use further to ­people below that level of risk. [This expanded use] is not supported by existing evidence.’

So will GPs stop giving out so many statins?

Despite the latest research, your GP may well start prescribing more. That’s because in a few months’ time the Joint British Societies will issue new guidelines.

This will mean that instead of calculating your risk of heart attack within ten years, your GP will calculate it for your lifetime and start treating you as soon as possible.

But as the doctors’ magazine Pulse recently warned, this means GPs faced with younger patients will inevitably reach for the statins. An editorial decries this ­‘latest step on the road to mass medicalisation’.

Are there other ways to protect my heart?

Last year Kausik Ray, professor of cardiovascular disease prevention at St George’s, University of London, published a review that, like the Cochrane Review, also found ­virtually no benefit from statins in primary prevention. He encourages patients to look at alternatives.

‘The data is very clear that statins can save some lives,’ he says. ‘But we need to get better at predicting who is going to benefit. ‘GPs have been pushed into a ­tick-box culture that means you get statins if you have certain risk ­factors,’ says Professor Ray. ‘But ideally you should discuss concerns, like how to handle long-term issues such as ­side-effects. Statins are unlikely to kill you but they can affect your quality of life.

‘You may want to try other treatments such as the B vitamin niacin, which has proved effective in trials recently. There are lots of options.’ Several trials have shown that niacin can bring down the ‘bad’ LDL cholesterol and raise the ‘good’ HDL. It does have a brief flushing effect on the skin, which some ­people find unpleasant.

SOURCE