Suppose you want to prove some point. But your little bit of data
doesn't quite prove it. Maybe more data would prove it. Watch out!
Collect more data, run the statistics again, did it work this time?
This is a slippery slope.
Suppose you are a hard worker, you're willing to run twenty tests, and
maybe one or two or three of them shows a significant result with
P<0.05. Could that happen with no real difference in your data? Of
course it could. On average more than 'could', it *will*! This is
the issue of the non-replicatable results, and the unpublished
non-results. There was a recent study saying most published results
don't replicate when someone else tries it again later. (One wonders
if that study was replicatable.) So you have a signficant result
maybe but run it a different way, by someone else, who maybe isn't
going to get so famous for the 'discovery', and it might well end up
different. How can we minimize that?
Some good practices will help people find truth instead of temporary
glory that becomes fake news.
- One practice is to show all your data. Putting your files on
the internet is not going to cost you anything if your department
already has a server. This means at least you should write down your
counts when you collect them.
- Another is to do your own replication. Collect the data you
need twice, once to seem to prove your point statistically and a
second time to check your reliability.
- Another is to graph the P values, and the two frequencies in
the two columns as your data count increases. Keep collecting data
until it looks pretty stable. Don't just stop at the moment it
crosses the line of P<0.05.
- Another is to decide in advance how much data you'll need to
prove your point, then collect just that much once. Then others might
thing at least that time was an honest experiment. But you might need
to make the previous graph shown above in order to make that
judgement. So if you're just starting to explore the question, expect
to estimate the relative frequencies in the first run, then pick a
target data count and a P level, then do it again to that count.
LI> Another is to think about one-tailed and two-tailed hypothesis
testing. Two tailed says, Is there a difference at all? One tailed
says, Is the difference the one I'm expecting to see. Once you
collect enough data for that, write your data down and don't use Tea
Lady which is for collecting data as you go, but instead use Oyvind
Langsrud's calculator where you just put in the totals.
Langsrud also has two-tailed and one-tailed results.
Yes, there's more! Scientists do their job not just by having ideas
and doing experiments, but also by finding things wrong with other
people's experiments. A good scientist is always trying to find
things wrong with their own experiments. It's not quite as bad as my
grizzled old Penn phonetics teacher Leigh Lisker said to me, The less
you say, the less likely you are to be wrong. Well, he was right,
sure, but still I say, just try to say things that are as true as you
can make them be, and be open to the process. Be a good scientist.