Robert Chung said:Then the recipe is a framework and a guideline within which you balance and adjust things to get
the best result. If you know what you're doing you get better results if you taste and adjust and
improvise. If you don't know what you're doing the dinner can end up a disaster and you'd have
been better off following the recipe. Having an exact cut-off is like following the recipe
exactly: it tends to protect your research findings from ending up as indigestible garbage. If you
know what you're doing then the p-level is just another parameter you consider when you're trying
to produce the best research.
Some scientists, no matter what they cook, it winds up tasting like fudge ...
As you suggest elsewhere, the difference between a marginally significant result with no mechanism
and the same set of data with a well justified mechanism is major, but not easy to express in
terms of p-values. That's subjective. So's science. The Bayesians have a word for this (I think
it's "Doh!"😉