Backtest blindness

Suppose you have to find a “brilliant” strategy for making money in the markets. You come up with a strategy you think will work. How do you “know” that it will work? One way is to perform backtests. This means you take past investment data, notably share prices, and pretend that you could use your strategy in the past and then see how much money you make.

So let’s suppose your strategy makes a lot of money. In fact, it seems to always make money. What could possibly go wrong? In this post, I will look at a few things that common sense dictate one should consider. A more academic investigation of backtesting will need to wait until a later post.

The curse of finite data

One problem is that you have only tested your strategy on a finite amount of past data. You only “know” it makes money if the future is exactly like this past. How likely is that?

A momentum strategy (involving buying companies whose shares have gone up and selling those whose shares have gone down) is one strategy that seems to perform very well over a very long time period. For instance if you were a quant at a trading desk in 2008 and you backtested such a strategy as far back as say 1940, you might have concluded you had a money-printing machine.

If, however, you had gone as far back as 1930 you would have seen this strategy could wipe out nearly all your capital in just a two month period. And had you implemented the strategy you would have experienced just that in 2009.

The lesson of this is not that you should have backtested all the way to 1930 (although you probably should have). You can only conceivably backtest about that far in any case – we simply do not have data going back much further. Of course a longer backtest is good, but you still have only a finite amount of data.

Markets can and do change

Markets do change, and sometimes quite abruptly. The by-now well-documented implied volatility smile observed in prices did not exist before 1987. Backtesting options strategies on pre-1987 data may not be very useful. The big crash in October seems to have permanently changed the way the market views options. But it could change again.

The problem with “successful” strategies

There is a lot of backtesting going on in financial institutions, I am quite certain. Lots of strategies will never see the light of day because they don’t produce high retrospective returns (they fail the backtest). However, the only strategies you are likely to come across as a potential investor are the ones that succeeded. These strategies have, by process of elimination, been optimised to produce excellent results in past conditions. This is collective data-mining.

These are the most misleading strategies. They are the most likely to disappoint because the past will not repeat itself exactly. This is much like trying to fit a curve to a number of data points – you can fit a curve that matches the data perfectly if you want, but it will have absolutely no predictive power.

Calibration

Calibrating a strategy can be a very dangerous thing to do. If your strategy has a few parameters and you try to find the ones that result in the most profit, you run into exactly the data-mining problem described above. The more parameters the more dangerous this becomes. It helps if you calibrate on one part of the data and test on a separate part. But this does not eliminate the problem – try enough strategies and you will find one that works both in and out of sample and completely fails in real life.

The problem with theories

You may think that if you come up with some brilliant idea, some model of market behaviour, that leads to a great strategy, all will be well. Not necessarily. The problem is that your idea is probably based on working with and observing markets and market data over a period. You are probably more likely to come up with a strategy that works well on past data merely because you know the past data better – even if only intuitively. This does not mean you have found a fundamental market law (perhaps the only fundamental market law is that any trading strategy will fail).

Strategies for which backtests do not work

You cannot backtest everything. Backtests assume you can take the past market prices as given and that you can trade at those prices. This only holds if the amounts you wish to trade are small compared to the volumes traded in the market. Thus backtesting will not work very well in illiquid markets and it will not work if you need buy or sell a large amount of stock that could potentially influence the market price. It is probably good practice to compare the volumes you wish to trade against the volumes actually traded in the past (noting that this changes from day to day).

How to keep the windscreen clear

One way to avoid at least some of the nasties of backtest blindness is to just conceive of a scenario in which your strategy would not make money (or better yet, in which it would lose a lot of money). It does not matter if it’s never happened. It doesn’t matter if it seems unlikely – you are bound to underestimate the probability of it occurring. Prepare for it anyway.

It is useful if whatever strategy you want to implement is based on some underlying theory – a theory that is likely to remain valid even if markets change. For instance, human behaviour is unlikely to change. If your strategy exploits fear and greed, it is more likely to succeed. However, this is no panacea. How do you know you’re actually exploiting human behaviour?

It helps if a strategy works in many markets – it is far more likely you are exploiting some fundamental human behaviour. However, more data is problematic if it gives you false confidence. More data is useful, but it does not negate the problems mentioned.

I admit I am not certain how to avoid all the pitfalls I mentioned above, at least not yet. But being aware of them is much better than not and that is a start.

Some references

• Barroso, P. & Santa-clara, P., 2012. Managing the Risk of Momentum. Business, (April), pp.1–26. Available at: http://ssrn.com/paper=2041429. Investing Answers, 2012. 

• Backtesting. Investing Answers . Available at: http://www.investinganswers.com/financial-dictionary/stock-market/backtesting-865 [Accessed November 17, 2012]. 

• Investopedia, 2012. Backtesting Definition. Investopedia. Available at: http://www.investopedia.com/terms/b/backtesting.asp#axzz2CNdPITKg [Accessed November 17, 2012]. 

• Wikipedia, 2012. Backtesting. Wikipedia. Available at: http://en.wikipedia.org/wiki/Backtesting [Accessed November 16, 2012]. (not a very good Wikipedia article)