Don't trust research, not even your own

The more research I do (and I have not done much) the less I trust other research, and the less I trust my own.  When you read a paper in finance you will invariably find significant results that support the author’s conclusions.  When I run regressions, my results are weak, sometimes contradicting my hypothesis sometimes confirming it.

Significance bias

Weak results don’t get published (and negative results, i.e. results that do not find evidence for an hypothesis don’t get published either).  Academics have an incentive to try things until they find something significant. Of course, they then only report the significant results. The problem is firstly that this is a form of data mining and secondly that there is information in the insignificant results that is now lost. Future researchers may well duplicate much of the work, not knowing it has already been tried.

The story bias

We like stories that fit together. In research (particularly research in finance) this can be problematic. At some point during your research (or beforehand) you will develop a theory, a kind of story puts all your results together like pieces of a puzzle. This will develop long before you have all your results. It’s a good thing to have a hypothesis before you start running tests on your data – it helps to avoid data mining. However, the danger is that as you look at your results, you discount the results that do not fit your story.  This may not even be conscious. It’s just the way your brain works.  The danger, then, is that you only report results that fit your story.  

As a researcher you are likely to have more results at your disposal than you can reasonably report in an article or thesis. You must condense, summarise, omit. More than that, you must give the appearance of a coherent structure, an evidence-based explanation of the underlying forces you have uncovered. You can’t just say, “well, it’s a bit of a mess.” Your story has to make sense. But sometimes the world is a mess and stories don’t work.

Self-examination

I wrote this post not because I found these tendencies in other researchers. I found their beginnings in myself. I have careful supervision that will prevent abuses. In theory peer review should uncover these things as well. In practice, I don’t trust the peer review system either. Thus this is a public reminder to myself: be wary of cherry-picking results to suit your prejudices.

Humble academics

(The following is based on my initial and brief impressions of the quality of academic writing in finance. I may change my mind later)

I’ve been reading through (the introductions) of very many articles in finance these past two weeks. The more I read, the more I realise that in finance the truth is a very murky prospect. In physics it seems like the truth is more stable (although physicists have a nasty habit of confusing their “theories” with reality. They seem to forget when their theories are shown to be wrong that they ever thought of them as Gospel).  But in finance, if you find two papers that agree, they probably share an author.

I am pretty sure that all these papers have one thing in common: they are all wrong. But every author is confident of his conclusions. References to why their results may be spurious are rare. Hardly ever do authors mention that their underlying assumptions are completely wrong – it seems standard to just rely on run-of-the-mill statistical methods, which I cannot believe take into account the wild randomness of the markets. Very few seem to care.

Academics in finance needs to be a little more humble. I think every paper should contain a disclaimer:

“The results in this paper are only valid under the assumptions of the methods used. These assumptions are almost certainly violated. The conclusions in this paper are disputed. Please do not confuse what is presented here with the truth.”

Some tips for academics:

• Write very clearly the underlying assumptions are – don’t just use methods without being very clear what it is they assume. 
• If you’re using a method outside of an area in which it is (proven) valid, write it in CAPS LOCK, because otherwise you’re a fraud, a charlatan.
• Show how the assumptions are violated (note I used “how” not “if”) – not just speculation, I want to see statistical tests and diagrams. 
•  Please reference everyone who disagrees with you. They’re not right either, but at least we know where to look for alternatives. 
• Stop being so sure of yourself.

Readers of anything in finance (of academic journals, of The Economist, etc.) should consider that anything can be challenged. There is no absolute truth. If there is, we cannot discover it, which amounts to the same thing. Live in a state of scepticism of everything you read. It isn’t fun – but the alternative, as Voltaire would say, is absurd.