Thread: Psychology Journal Bans Significance Testing

Red herring. No one actually says that p>0.05 instantly makes the results "useless". The required significance level always depends on context, and the judgment of experts. For example, in quantum physics experiments you might say that any p>0.001 is not significant. In psychology, perhaps p>0.1 is more appropriate, but it depends on other factors.

Concur with cmind. There may be valid reasons for discouraging use of the p-value, but the 0.05 thing isn't one of them, and you'll notice that it wasn't a reason cited by the article. A p-value of 0.051 would be interpreted pretty much like a p-value of 0.049. Nobody really thinks there's a binary cut-off.

I think my issue about the 0.05 thing has to do with how research gets published. What I mean is, if somebody ends up with p>0.05 in their statistical analysis, how often are their results ignored? I don't mean by the wider community, I mean by the researchers themselves. Negative results aren't published nearly as much as positive results. I should've been more clear.

And yes, I know the SBM article (or the article cited within it) doesn't mention it. The thoughts in the OP were my own.

I think a bigger issue, especially in the softer sciences, is the failure of some researchers to differentiate between observational variables and experimental variables. Usually, this means treating the former as if it was the latter. The difference between them is that an observation can only tell you correlations, whereas an experiment can tell you causes.

A recent example would be studies that correlate pot smoking with depression or schizophrenia. The incidence of mental illness at any given time is merely an observation, not an experiment. An experiment would involve telling one group of people to never smoke pot, and another group to smoke it every day, and watch what happens over a period of several years. This is obviously unethical and can never be done. So the researches pull a fast one, and treat the observation of mental illness as if they did do the experiment. Not surprisingly, they find strong correlation between mental illness and pot smoking, and then say that the marijuana caused the illness. This is an egregious abuse of statistics, and is borderline evil, considering the political consequences of such papers.