Don't trust research, not even your own

The more research I do (and I have not done much) the less I trust other research, and the less I trust my own.  When you read a paper in finance you will invariably find significant results that support the author’s conclusions.  When I run regressions, my results are weak, sometimes contradicting my hypothesis sometimes confirming it.

Significance bias

Weak results don’t get published (and negative results, i.e. results that do not find evidence for an hypothesis don’t get published either).  Academics have an incentive to try things until they find something significant. Of course, they then only report the significant results. The problem is firstly that this is a form of data mining and secondly that there is information in the insignificant results that is now lost. Future researchers may well duplicate much of the work, not knowing it has already been tried.

The story bias

We like stories that fit together. In research (particularly research in finance) this can be problematic. At some point during your research (or beforehand) you will develop a theory, a kind of story puts all your results together like pieces of a puzzle. This will develop long before you have all your results. It’s a good thing to have a hypothesis before you start running tests on your data – it helps to avoid data mining. However, the danger is that as you look at your results, you discount the results that do not fit your story.  This may not even be conscious. It’s just the way your brain works.  The danger, then, is that you only report results that fit your story.  

As a researcher you are likely to have more results at your disposal than you can reasonably report in an article or thesis. You must condense, summarise, omit. More than that, you must give the appearance of a coherent structure, an evidence-based explanation of the underlying forces you have uncovered. You can’t just say, “well, it’s a bit of a mess.” Your story has to make sense. But sometimes the world is a mess and stories don’t work.

Self-examination

I wrote this post not because I found these tendencies in other researchers. I found their beginnings in myself. I have careful supervision that will prevent abuses. In theory peer review should uncover these things as well. In practice, I don’t trust the peer review system either. Thus this is a public reminder to myself: be wary of cherry-picking results to suit your prejudices.