Skip to content
ResearchBeyond

What significance testing does and doesn't tell you

A significant result means a difference is unlikely to be down to chance. That is all it means.

A significant result means a difference is unlikely to be down to sampling chance alone. That is all it means.

It doesn’t say the difference is large. With a big enough sample, a tiny gap is significant. It doesn’t say the difference matters to your business. And if you test fifty comparisons, a few will come out significant by chance.

Treat it as a filter, not a verdict. Look at the size of the difference, decide in advance which comparisons matter, and ask whether the finding fits what you heard in interviews.