The anonymous referee

In academic articles, you sometimes come across a paragraph that makes no sense (even on a second or third reading). Very often you will find, attached to this paragraph, a footnote, which says something like the following

“We thank the anonymous referee for drawing our attention to this.”

To the anonymous referee, that is exactly what is being said. But to anyone else reading the article it translates as

“We had to put this in our article in order to get it published because the anonymous referee is an idiot and insisted on it.”

One can then proceed to read the paper, ignoring that paragraph. This seems wrong to me. On the one hand I understand the trade-off: no publication vs make one little, albeit nonsensical, change. I would like to think I have more honour, more backbone than that, but I probably do not.

I think the anonymous peer review system is flawed. And it appears I’m not the only that thinks so.

Peer review is currently (at least in some fields and journals, not all though) double blind. In theory the referee does not know whose paper they are reviewing and the author does not know who the referee is (in practice, I think, anonymity is hardly guaranteed, especially in fields where there are a small number of specialists. Your writing, the references you choose (especially your own papers) can give you away).

It is one side of this double blindness that bothers me – that referees are anonymous. The other side may have flaws too, but at least it should prevent a paper being accepted merely because it is written by a bigshot academic.

Referee anonymity absolves journals of the responsibility of explaining their choice of referee (if they choose a referee obviously against the line of research or too obviously biased for it). It also means referees are not held accountable for their reviews. They may not take the process seriously and out of sheer laziness rather than malice block good research or let bad research pass.

I think that if referees are made known it will allow for greater dialogue. Referees can be challenged. Their reputations depend on being thorough. The feedback process may in fact lead to better research.

I think I, like one Dr Bertrand Meyer (see below), will always insist on signing my reviews (if I am ever in the position to review work for publication). My reputation is important to me. I want my reviews to reflect on my reputation (otherwise I will not take them seriously) and I want them to be thorough and thoughtful (otherwise they will reflect badly on my reputation).

• Meyer, B. (2013). why I sign my referee reports. Retrieved January 29, 2013, from http://se.ethz.ch/~meyer/publications/online/whysign/
• Watson, R. (1985). Anonymous Referees and Blind Refereeing. Proceedings and Addresses of the American Philosophical Association, 85(5), 755–757. Retrieved from http://www.jstor.org/stable/10.2307/3131683
• Wikipedia. (2013). Peer review. Wikipedia. Retrieved January 31, 2013, from http://en.wikipedia.org/wiki/Peer_review

Review: Models. Behaving. Badly

This book was written by the famous quant “Emanuel Derman”, whom I mentioned in one of my blog posts before when I commented on the Financial Modeller’s Manifesto.  I was expecting a lot from this book, I admit. And I was disappointed. That is not to say that the book did not contain valuable insight, but I was hoping for more. For a book inspired by the financial crisis, it has precious little to say about it.

Not really about finance: The very long preamble

If you were hoping to read a book about finance (or at least financial models) with some references to other material for diversion (as I was) you will be disappointed.  Most of the book hardly even mentions finance. Instead it deals with the Emanuel’s (admittedly not uninteresting) view of models in physics,  society (such as during the apartheid era) and Spinozan philosophy. The point of this, I think, was to illustrate in a more general setting the idea of a model or a theory. But given that the book is portrayed as being firstly about “Wall Street” it feels a bit like fluff.

There are some autobiographical passages about Dermans life in South Africa. I found these very interesting, but they added little value to the goal of the book. The point Derman was trying to make (that the models used in apartheid South Africa failed) could have been made in much less space.  But then the book would have been even shorter than it already is. I hardly think anyone who buys the book would be truly interested in reading about Spinoza’s theory of emotions (as interesting it might be philosophically). I certainly hoped the financial stuff would come soon.

One would have expected to get at least a good explanation of how models were used during the financial crisis and how they failed. Instead, the links that Derman makes with his descriptions of some basic financial models and the financial crisis are superficial at best. If you want insight into this part of the financial crisis, you must go elsewhere.  Early on in the book Derman laments what had happened during the crisis and before it: “decline of manufacturing; the ballooning of the financial sector; that sector’s capture of the regulatory system; ceaseless stimulus whenever the economy has waivered; tax-payer-funded bailouts…” It’s a very long list and not one item on it is treated in the book. We are told that model failure was the cause – we are never given any more insight than that.

The value of commonsense

I have been quite critical thus far, but the book does add value. There is a distinction between models used in physics, which are accurate, and those used in finance which are, at best, sometimes useful. The latter often treat people as if they are just particles or objects, which they are not. Derman calls this “pragmamorphism”. Financial models always leave out something important. The admonition to always use common sense is valid. However, I was hoping to come away with more insight than that. Perhaps that’s all there is, really.

Models and theories and facts – Derman does the unforgivable

Central to the book is the distinction between “models”, which are based on analogy, and
“theories” which attempt to describe the real world without analogy. Essentially, physics works with theories (mostly) and finance works exclusively with models. This is a useful distinction – though I am not convinced that the two categories are not instead two extremes of a continuum of models. However, as far as thinking about modelling goes, I believe it is very valuable.

Dr Derman goes one step further though, doing something I find unforgivable.  He claims that a “correct” theory becomes a fact. Physics models that say there are electrons and that they behave in certain ways are the truth. I do not think Dr Derman actually thinks this – because to do so would be to disavow even the possibility of a theory being overturned, replaced by something better. And we have seen it done: Newton’s laws, “confirmed” to be accurate for hundreds of years turned out to be a poor description of reality once you started looking at things moving near the speed of light.

Physics uses mathematics and mathematics is not and will never be the real world – though it is the most useful tool we have for describing the world. In science (all of science, including physics) we can only ever say this: IF my model or theory is correct then we would expect certain observations in the real world.

Science can never confirm a theory to be correct. Theories that are considered “facts” are just the ones that have not yet been proven to be wrong. I think that a better theory than general relativity or quantum electrodynamics may come along – it may only bring incremental changes or it may bring a revolution in the way we think about the world. But it is the way we think about the world that changes, not the world.

Verdict

I must, if I am kind, conclude that Derman’s book tries to do a little too much (or, if I am unkind, that it tries to do too little and pads it with fluff): it wants to be philosophy, biography, essay and social commentary. It does none of these particularly well. 

Reference

Derman, E., 2012. Models.Behaving.Badly.: Why Confusing Illusion with Reality Can Lead to Disaster, on Wall Street and in Life, Free Press. Available at: http://www.amazon.com/Models-Behaving-Badly-Confusing-Illusion-Reality-Disaster/dp/1439164991 [Accessed January 27, 2013].

Humble academics

(The following is based on my initial and brief impressions of the quality of academic writing in finance. I may change my mind later)

I’ve been reading through (the introductions) of very many articles in finance these past two weeks. The more I read, the more I realise that in finance the truth is a very murky prospect. In physics it seems like the truth is more stable (although physicists have a nasty habit of confusing their “theories” with reality. They seem to forget when their theories are shown to be wrong that they ever thought of them as Gospel).  But in finance, if you find two papers that agree, they probably share an author.

I am pretty sure that all these papers have one thing in common: they are all wrong. But every author is confident of his conclusions. References to why their results may be spurious are rare. Hardly ever do authors mention that their underlying assumptions are completely wrong – it seems standard to just rely on run-of-the-mill statistical methods, which I cannot believe take into account the wild randomness of the markets. Very few seem to care.

Academics in finance needs to be a little more humble. I think every paper should contain a disclaimer:

“The results in this paper are only valid under the assumptions of the methods used. These assumptions are almost certainly violated. The conclusions in this paper are disputed. Please do not confuse what is presented here with the truth.”

Some tips for academics:

• Write very clearly the underlying assumptions are – don’t just use methods without being very clear what it is they assume. 
• If you’re using a method outside of an area in which it is (proven) valid, write it in CAPS LOCK, because otherwise you’re a fraud, a charlatan.
• Show how the assumptions are violated (note I used “how” not “if”) – not just speculation, I want to see statistical tests and diagrams. 
•  Please reference everyone who disagrees with you. They’re not right either, but at least we know where to look for alternatives. 
• Stop being so sure of yourself.

Readers of anything in finance (of academic journals, of The Economist, etc.) should consider that anything can be challenged. There is no absolute truth. If there is, we cannot discover it, which amounts to the same thing. Live in a state of scepticism of everything you read. It isn’t fun – but the alternative, as Voltaire would say, is absurd.