The anonymous referee

In academic articles, you sometimes come across a paragraph that makes no sense (even on a second or third reading). Very often you will find, attached to this paragraph, a footnote, which says something like the following

“We thank the anonymous referee for drawing our attention to this.”

To the anonymous referee, that is exactly what is being said. But to anyone else reading the article it translates as

“We had to put this in our article in order to get it published because the anonymous referee is an idiot and insisted on it.”

One can then proceed to read the paper, ignoring that paragraph. This seems wrong to me. On the one hand I understand the trade-off: no publication vs make one little, albeit nonsensical, change. I would like to think I have more honour, more backbone than that, but I probably do not.

I think the anonymous peer review system is flawed. And it appears I’m not the only that thinks so.

Peer review is currently (at least in some fields and journals, not all though) double blind. In theory the referee does not know whose paper they are reviewing and the author does not know who the referee is (in practice, I think, anonymity is hardly guaranteed, especially in fields where there are a small number of specialists. Your writing, the references you choose (especially your own papers) can give you away).

It is one side of this double blindness that bothers me – that referees are anonymous. The other side may have flaws too, but at least it should prevent a paper being accepted merely because it is written by a bigshot academic.

Referee anonymity absolves journals of the responsibility of explaining their choice of referee (if they choose a referee obviously against the line of research or too obviously biased for it). It also means referees are not held accountable for their reviews. They may not take the process seriously and out of sheer laziness rather than malice block good research or let bad research pass.

I think that if referees are made known it will allow for greater dialogue. Referees can be challenged. Their reputations depend on being thorough. The feedback process may in fact lead to better research.

I think I, like one Dr Bertrand Meyer (see below), will always insist on signing my reviews (if I am ever in the position to review work for publication). My reputation is important to me. I want my reviews to reflect on my reputation (otherwise I will not take them seriously) and I want them to be thorough and thoughtful (otherwise they will reflect badly on my reputation).

• Meyer, B. (2013). why I sign my referee reports. Retrieved January 29, 2013, from http://se.ethz.ch/~meyer/publications/online/whysign/
• Watson, R. (1985). Anonymous Referees and Blind Refereeing. Proceedings and Addresses of the American Philosophical Association, 85(5), 755–757. Retrieved from http://www.jstor.org/stable/10.2307/3131683
• Wikipedia. (2013). Peer review. Wikipedia. Retrieved January 31, 2013, from http://en.wikipedia.org/wiki/Peer_review

Review: Models. Behaving. Badly

This book was written by the famous quant “Emanuel Derman”, whom I mentioned in one of my blog posts before when I commented on the Financial Modeller’s Manifesto.  I was expecting a lot from this book, I admit. And I was disappointed. That is not to say that the book did not contain valuable insight, but I was hoping for more. For a book inspired by the financial crisis, it has precious little to say about it.

Not really about finance: The very long preamble

If you were hoping to read a book about finance (or at least financial models) with some references to other material for diversion (as I was) you will be disappointed.  Most of the book hardly even mentions finance. Instead it deals with the Emanuel’s (admittedly not uninteresting) view of models in physics,  society (such as during the apartheid era) and Spinozan philosophy. The point of this, I think, was to illustrate in a more general setting the idea of a model or a theory. But given that the book is portrayed as being firstly about “Wall Street” it feels a bit like fluff.

There are some autobiographical passages about Dermans life in South Africa. I found these very interesting, but they added little value to the goal of the book. The point Derman was trying to make (that the models used in apartheid South Africa failed) could have been made in much less space.  But then the book would have been even shorter than it already is. I hardly think anyone who buys the book would be truly interested in reading about Spinoza’s theory of emotions (as interesting it might be philosophically). I certainly hoped the financial stuff would come soon.

One would have expected to get at least a good explanation of how models were used during the financial crisis and how they failed. Instead, the links that Derman makes with his descriptions of some basic financial models and the financial crisis are superficial at best. If you want insight into this part of the financial crisis, you must go elsewhere.  Early on in the book Derman laments what had happened during the crisis and before it: “decline of manufacturing; the ballooning of the financial sector; that sector’s capture of the regulatory system; ceaseless stimulus whenever the economy has waivered; tax-payer-funded bailouts…” It’s a very long list and not one item on it is treated in the book. We are told that model failure was the cause – we are never given any more insight than that.

The value of commonsense

I have been quite critical thus far, but the book does add value. There is a distinction between models used in physics, which are accurate, and those used in finance which are, at best, sometimes useful. The latter often treat people as if they are just particles or objects, which they are not. Derman calls this “pragmamorphism”. Financial models always leave out something important. The admonition to always use common sense is valid. However, I was hoping to come away with more insight than that. Perhaps that’s all there is, really.

Models and theories and facts – Derman does the unforgivable

Central to the book is the distinction between “models”, which are based on analogy, and
“theories” which attempt to describe the real world without analogy. Essentially, physics works with theories (mostly) and finance works exclusively with models. This is a useful distinction – though I am not convinced that the two categories are not instead two extremes of a continuum of models. However, as far as thinking about modelling goes, I believe it is very valuable.

Dr Derman goes one step further though, doing something I find unforgivable.  He claims that a “correct” theory becomes a fact. Physics models that say there are electrons and that they behave in certain ways are the truth. I do not think Dr Derman actually thinks this – because to do so would be to disavow even the possibility of a theory being overturned, replaced by something better. And we have seen it done: Newton’s laws, “confirmed” to be accurate for hundreds of years turned out to be a poor description of reality once you started looking at things moving near the speed of light.

Physics uses mathematics and mathematics is not and will never be the real world – though it is the most useful tool we have for describing the world. In science (all of science, including physics) we can only ever say this: IF my model or theory is correct then we would expect certain observations in the real world.

Science can never confirm a theory to be correct. Theories that are considered “facts” are just the ones that have not yet been proven to be wrong. I think that a better theory than general relativity or quantum electrodynamics may come along – it may only bring incremental changes or it may bring a revolution in the way we think about the world. But it is the way we think about the world that changes, not the world.

Verdict

I must, if I am kind, conclude that Derman’s book tries to do a little too much (or, if I am unkind, that it tries to do too little and pads it with fluff): it wants to be philosophy, biography, essay and social commentary. It does none of these particularly well. 

Reference

Derman, E., 2012. Models.Behaving.Badly.: Why Confusing Illusion with Reality Can Lead to Disaster, on Wall Street and in Life, Free Press. Available at: http://www.amazon.com/Models-Behaving-Badly-Confusing-Illusion-Reality-Disaster/dp/1439164991 [Accessed January 27, 2013].

Humble academics

(The following is based on my initial and brief impressions of the quality of academic writing in finance. I may change my mind later)

I’ve been reading through (the introductions) of very many articles in finance these past two weeks. The more I read, the more I realise that in finance the truth is a very murky prospect. In physics it seems like the truth is more stable (although physicists have a nasty habit of confusing their “theories” with reality. They seem to forget when their theories are shown to be wrong that they ever thought of them as Gospel).  But in finance, if you find two papers that agree, they probably share an author.

I am pretty sure that all these papers have one thing in common: they are all wrong. But every author is confident of his conclusions. References to why their results may be spurious are rare. Hardly ever do authors mention that their underlying assumptions are completely wrong – it seems standard to just rely on run-of-the-mill statistical methods, which I cannot believe take into account the wild randomness of the markets. Very few seem to care.

Academics in finance needs to be a little more humble. I think every paper should contain a disclaimer:

“The results in this paper are only valid under the assumptions of the methods used. These assumptions are almost certainly violated. The conclusions in this paper are disputed. Please do not confuse what is presented here with the truth.”

Some tips for academics:

• Write very clearly the underlying assumptions are – don’t just use methods without being very clear what it is they assume. 
• If you’re using a method outside of an area in which it is (proven) valid, write it in CAPS LOCK, because otherwise you’re a fraud, a charlatan.
• Show how the assumptions are violated (note I used “how” not “if”) – not just speculation, I want to see statistical tests and diagrams. 
•  Please reference everyone who disagrees with you. They’re not right either, but at least we know where to look for alternatives. 
• Stop being so sure of yourself.

Readers of anything in finance (of academic journals, of The Economist, etc.) should consider that anything can be challenged. There is no absolute truth. If there is, we cannot discover it, which amounts to the same thing. Live in a state of scepticism of everything you read. It isn’t fun – but the alternative, as Voltaire would say, is absurd. 

Out the tragedy... fun

It is interesting how a tragedy impresses itself upon the human collective conscience, how it is transformed, in rather unexpected ways. From the holocaust we have books, films, plays. From the financial crisis and its various extensions of financial meltdown we have gotten a Margin Call, a movie with a rather loose grip on reality  (as is the case with most movies) and, I now find, a game called Market Meltdown.

What is it?

The game appears to be based on the antics of a number of rogue traders. In Market Meltdown the players are traders who must make ever bigger bets (as the money they owe increases) in order to stay afloat. The losers are those who go bust first.

Just some fun

It is likely the board game is just good fun, amoral. That is what it is intended to be. It is hard to tell if the game will succeed, whether perhaps people will find it distasteful. But if kids can play computer games pretending to be evil overlords, why not have families play games pretending to be reckless traders. It could be used as an exercise in moral instruction – don’t do this in real life.

But in our hearts

The fact is that there is something about these rogue traders that captures our hearts. Sure, we’re angry with them for being heartless human beings, extensions of a merciless financial sector. But it’s not our money they lost. What a thrill it must be to place such huge bets, to waver between being ignominious Kweku Adoboli or glorious George Soros. (Admittedly Soros was no rogue trader, but his bets were massive and if he were wrong his ignominy would have been as great, if not necessarily prosecutable).

Social commentary

In some sense, the game, with its simplified rules, its exaggerated nature, is a kind of mockery. What were they thinking? We give them, those rogue traders, immortality, not by their reputation, but by frequently reliving their defeat, and laughing at it. Perhaps that, too, is a kind of punishment.

Out of sight

Despite the financial crisis’s impact on people, it being discussed everywhere. Despite the fate of Greece and the entire Eurozone hanging in the balance, I do not think that it is a ripe source of creative inspiration. There are a number of other games inspired by the crisis. Still, I would not imagine that many books of fiction will be written about these times once they are past, that many more films will be made (though there will undoubtedly be some). The halls of finance are just too hard for people to imagine, the dealing and wheedling too obscure, the language too foreign, with certain obvious exceptions, it’s just not exciting enough (although I may be wrong).

It is a pity. Perhaps if we could bring the world of finance to the hearts of people, we could place some heart in finance.

Some references

The game:

• Clarendon Games, 2012a. Market Meltdown. Available at: http://www.clarendongames.com/product.php?xProd=19 [Accessed December 29, 2012]. Clarendon Games, 2012b. 
• Market Meltdown Intro. Available at: http://www.youtube.com/watch?v=lq_AdyyiQx4 [Accessed December 29, 2012]. 

The Economist's take:

• The Economist, 2012. Financial board games: Playing the markets. The Economist. Available at: http://www.economist.com/blogs/prospero/2012/12/financial-board-games [Accessed December 29, 2012]. 

George Soros: 

• Wikipedia, 2012a. George Soros. Wikipedia. Available at: http://en.wikipedia.org/wiki/Soros [Accessed December 29, 2012]. 

Rogue trader Kweku Adoboli (he is just the most recent):

•  Wikipedia, 2012b. Kweku Adoboli - Wikipedia, the free encyclopedia. Wikipedia. Available at: http://en.wikipedia.org/wiki/Kweku_Adoboli [Accessed December 29, 2012].

Backtest blindness

Suppose you have to find a “brilliant” strategy for making money in the markets. You come up with a strategy you think will work. How do you “know” that it will work? One way is to perform backtests. This means you take past investment data, notably share prices, and pretend that you could use your strategy in the past and then see how much money you make.

So let’s suppose your strategy makes a lot of money. In fact, it seems to always make money. What could possibly go wrong? In this post, I will look at a few things that common sense dictate one should consider. A more academic investigation of backtesting will need to wait until a later post.

The curse of finite data

One problem is that you have only tested your strategy on a finite amount of past data. You only “know” it makes money if the future is exactly like this past. How likely is that?

A momentum strategy (involving buying companies whose shares have gone up and selling those whose shares have gone down) is one strategy that seems to perform very well over a very long time period. For instance if you were a quant at a trading desk in 2008 and you backtested such a strategy as far back as say 1940, you might have concluded you had a money-printing machine.

If, however, you had gone as far back as 1930 you would have seen this strategy could wipe out nearly all your capital in just a two month period. And had you implemented the strategy you would have experienced just that in 2009.

The lesson of this is not that you should have backtested all the way to 1930 (although you probably should have). You can only conceivably backtest about that far in any case – we simply do not have data going back much further. Of course a longer backtest is good, but you still have only a finite amount of data.

Markets can and do change

Markets do change, and sometimes quite abruptly. The by-now well-documented implied volatility smile observed in prices did not exist before 1987. Backtesting options strategies on pre-1987 data may not be very useful. The big crash in October seems to have permanently changed the way the market views options. But it could change again.

The problem with “successful” strategies

There is a lot of backtesting going on in financial institutions, I am quite certain. Lots of strategies will never see the light of day because they don’t produce high retrospective returns (they fail the backtest). However, the only strategies you are likely to come across as a potential investor are the ones that succeeded. These strategies have, by process of elimination, been optimised to produce excellent results in past conditions. This is collective data-mining.

These are the most misleading strategies. They are the most likely to disappoint because the past will not repeat itself exactly. This is much like trying to fit a curve to a number of data points – you can fit a curve that matches the data perfectly if you want, but it will have absolutely no predictive power.

Calibration

Calibrating a strategy can be a very dangerous thing to do. If your strategy has a few parameters and you try to find the ones that result in the most profit, you run into exactly the data-mining problem described above. The more parameters the more dangerous this becomes. It helps if you calibrate on one part of the data and test on a separate part. But this does not eliminate the problem – try enough strategies and you will find one that works both in and out of sample and completely fails in real life.

The problem with theories

You may think that if you come up with some brilliant idea, some model of market behaviour, that leads to a great strategy, all will be well. Not necessarily. The problem is that your idea is probably based on working with and observing markets and market data over a period. You are probably more likely to come up with a strategy that works well on past data merely because you know the past data better – even if only intuitively. This does not mean you have found a fundamental market law (perhaps the only fundamental market law is that any trading strategy will fail).

Strategies for which backtests do not work

You cannot backtest everything. Backtests assume you can take the past market prices as given and that you can trade at those prices. This only holds if the amounts you wish to trade are small compared to the volumes traded in the market. Thus backtesting will not work very well in illiquid markets and it will not work if you need buy or sell a large amount of stock that could potentially influence the market price. It is probably good practice to compare the volumes you wish to trade against the volumes actually traded in the past (noting that this changes from day to day).

How to keep the windscreen clear

One way to avoid at least some of the nasties of backtest blindness is to just conceive of a scenario in which your strategy would not make money (or better yet, in which it would lose a lot of money). It does not matter if it’s never happened. It doesn’t matter if it seems unlikely – you are bound to underestimate the probability of it occurring. Prepare for it anyway.

It is useful if whatever strategy you want to implement is based on some underlying theory – a theory that is likely to remain valid even if markets change. For instance, human behaviour is unlikely to change. If your strategy exploits fear and greed, it is more likely to succeed. However, this is no panacea. How do you know you’re actually exploiting human behaviour?

It helps if a strategy works in many markets – it is far more likely you are exploiting some fundamental human behaviour. However, more data is problematic if it gives you false confidence. More data is useful, but it does not negate the problems mentioned.

I admit I am not certain how to avoid all the pitfalls I mentioned above, at least not yet. But being aware of them is much better than not and that is a start.

Some references

• Barroso, P. & Santa-clara, P., 2012. Managing the Risk of Momentum. Business, (April), pp.1–26. Available at: http://ssrn.com/paper=2041429. Investing Answers, 2012. 

• Backtesting. Investing Answers . Available at: http://www.investinganswers.com/financial-dictionary/stock-market/backtesting-865 [Accessed November 17, 2012]. 

• Investopedia, 2012. Backtesting Definition. Investopedia. Available at: http://www.investopedia.com/terms/b/backtesting.asp#axzz2CNdPITKg [Accessed November 17, 2012]. 

• Wikipedia, 2012. Backtesting. Wikipedia. Available at: http://en.wikipedia.org/wiki/Backtesting [Accessed November 16, 2012]. (not a very good Wikipedia article)

Review: The Big Short

The Big Short by Michael Lewis is a very good book, thoroughly entertaining. The language is simple, but all the concepts are sufficiently explained to make sense to non-financial readers, without leaving out too much. It also cleared up some issues I have been unable to get a grip on despite reading sporadically about the crisis for years. I do recommend that you read this book. But if you do, beware of a couple of things.

It’s a story

The books reads like a story, which is what makes it so entertaining. But it is also exactly there that its danger lies. It is centred on a handful of individuals that made money from the subprime crisis, who, in essence, predicted it and took investment positions to profit from it.  It contains many personal elements of the lives of these people, explaining how they moved through life and how they decided to place their bets against the system.  Their lives are very interesting, at least Lewis manages to make them appear so.

Don’t trust a story

Storytelling, though useful, is dangerous. It makes you forget how messy life really is. Everything is put into plots and subplots, everything heads toward the ending, in this case, a financial collapse. It has the illusion of inevitability. Never believe that.  As smart as these people were, as thorough as their research was, nothing about their success was inevitable. They were lucky. I do not mean to say the odds were not in their favour – they looked at information most others ignored, they saw things others did not see. But they could (or rather, should not) have been certain. A difference in timing, a slight change in the economy, stimulus here or not there, and we could have seen a different set of winners or losers.

There is a survivorship bias in the book. (This is also a problem with stories – usually they focus on the people who succeed, sometimes on the ones who fail horribly, never the ones in-between). We hear only from the people who made it. How they happened to make their fortune. We don’t hear about the people in similar situations with similar intellects who did not. We are to presume they did not exist.

As I read and I felt the suspense of the coming crash (and voyeuristic  exhilaration at the heaps and heaps of money the protagonists would make) I too felt, perhaps I can make money too, perhaps I could also be a great investor. Of course these people were not really investors, they were speculators betting on the crash of a system (albeit, probably with the odds in their favour).  They were right, at least partly because they were lucky enough to stumble on the right information at the right time.

Disclaimers

Perhaps, if there is anything to learn from the crisis and from the book, it is that you are far more likely to be the sucker who misunderstands everything (in this case almost all of Wall Street) than anything else. Humility is your ally.

I do not mean to say there is anything wrong with the book. It has its uses and its limitations. But perhaps it should contain a disclaimer about the danger of stories.

Knight in soiled armour

Last week the markets experienced another little jolt, similar to the flash crash in 2010. A computer glitch at Knight Capital caused its systems to send out incorrect orders, causing huge price swings in stocks and driving the company nearly to bankruptcy.

Bugs

There is now talk of changing regulations (again) and everyone wonders what Knight Capital did wrong. They had a bug. I cannot be certain, but my guess is that whatever software they were using had been tested. But with sophisticated systems bugs always slip through (it’s inevitable). Unluckily for Knight their particular bug caused a lot of trouble and damaged the company’s reputation, possibly irreparably.

It’s (almost) all in the mind

What is interesting is that the company is set to survive. Knight was bailed out not by government, but by a handful of its competitors – perhaps they know this could just as easily have happened to them. They were not just being helpful, though. They saw an opportunity to buy a large stake of a good business (potentially good, in any case) very cheaply. The major obstacles to Knight’s continued survival were a lack of capital and loss of confidence. Both were addressed by the bailout. The very fact that these companies were willing to fund Knight will give the market assurance that the business was worth saving.

Reputations

This bailout comes at a price to current shareholders (who are the ones who should pay – not taxpayers and not clients) whose holdings are diluted. Perhaps, slowly, clients will return. The key reason to think this may happen is the reputation of Knight’s CEO, Thomas Joyce, which seems to have been both battered and uplifted in the debacle. Unlike Bob Diamond, CEO of Barclays and other banks’ top brass, Joyce’s integrity is not in question. He has been called a hero for managing to get hold of the much needed financing. Knight also, it seems, absorbed most of the losses – shielding their clients.

What is in question is Knight’s risk management and software testing. And after an incident like this, I think this is liable to become too strict rather than too relaxed. They cannot afford another incident – that would almost certainly end them, if not through a direct loss, then due to a loss of confidence.

Operational risk 

Knight’s problems highlight, once again, that the major risk in business, any business, is operational. It is unpredictable, its costs can be little or gigantic. Even with good risk management procedures (which are a must – and there is no reason to believe Knight’s were not adequate) mistakes will be made. The regulatory reactions to this will (probably) be firstly to increase the amount of capital that companies need to hold so they can absorb operational losses, secondly to mandate more stringent risk management, and thirdly to demand more detailed reports to the regulator (in this case the SEC). All of these things have their costs.

People will, of course, be very interested in knowing what exactly caused the software malfunction and how. And what Knight will be doing to prevent it from happening again. But what the error was hardly matters – it was random. Next time it will be something else. Hopefully responses will focus less on the specific nature of the problem that arose and more on the general nature of operational risk, which has a tendency to pop up in unexpected places.

References

• Kisling, W., & Mehta, N. (2012). Joyce Puts Knight Survival Over Shares in Rescue Deal. Bloomberg. Retrieved August 7, 2012, from http://www.bloomberg.com/news/2012-08-06/joyce-puts-knight-survival-over-shares-forging-400-million-deal.html
• Pratley, N. (2012). Knight Capital’s computer “glitch” shows dangers of desire for faster trading. The Guardian. Retrieved August 7, 2012, from http://www.guardian.co.uk/business/nils-pratley-on-finance/2012/aug/06/knight-capital-computer-glitch-trading?newsfeed=true
• Reuters. (2012). Knight Capital handed $400m lifeline after trading debacle. The Guardian. Retrieved August 7, 2012, from http://www.guardian.co.uk/business/2012/aug/06/knight-capital-400m-lifeline
• Sapa-AP. (2012). Knight Capital’s $440m computer glitch. Times Live. Retrieved August 7, 2012, from http://www.timeslive.co.za/scitech/2012/08/03/knight-capital-s-440m-computer-glitch
• The Economist. (2012). Desperate times. The Economist. Retrieved August 7, 2012, from http://www.economist.com/blogs/schumpeter/2012/08/knight-capital?fsrc=scn/fb/wl/bl/desperatetimes
• Touryalai, H. (2012). Knight Capital: The Ideal Way To Screw Up On Wall Street. Forbes. Retrieved August 7, 2012, from http://www.forbes.com/sites/halahtouryalai/2012/08/06/knight-capital-the-ideal-way-to-screw-up-on-wall-street/