10 Reasons why back-testing is pointless

I am always interested in why my endeavours fail, but i usually try to do something about it. Often the knowledge so gained sets me upon a different, hopefully profitable path.

Recently, one of my bots had some issues which made me decide to use "better data" for my backtesting and optimisation. The following is a bullet-point list of the results.

1. You are not using your own broker's live server data:

- It comes from a different broker than the one on which you will forward test, and so

will almost certainly have (even slightly) different prices from that which you would have experienced had you been connected to your broker's platform. Prices differ also between demo and live, and between historic timeframe data (*see footnote*)

2. You are using your own broker's (tick) data, but the copy that you have is not complete and/or accurate:

- This would probably be the most accurate way to test a bot, but relies on you receiving/recording every tick in a timely fashion. This may not be possible due to net latency.

3. You have filtered out duplicate ticks

- Duplicate ticks may have an effect on your bot.

4. You are using a constant spread:

- Spreads are usually only constant when the market is closed.

5. You are using a constant tick value:

- Tick values are sometimes not constant, e.g. when trading a cross pair.

6. Your bot behaves differently in live than in optimisation/testing mode:

- Some platforms provide a way of the software determining if it is "test" mode, e.g. the IsOptimisation() and IsTesting() Metatrader functions. These allow you to legitimately bypass unwanted code (e.g. some console output) but may also cause other side effects.

7. You assume market memory where there is none.

- If the random-walk theory is correct (not my opinion), then technical analysis of any kind is useless.

8. Your bot is over-tuned (curve-fitted) to the price data:

- Some analysts insist a bot must have been back-tested using many years of data, and is only "useful" if it would have been consistently profitable over that period.

9. Your bot is under-tuned (not curve-fitted) to the price data:

- Some analysts believe that regular tuning is necessary to allow the bot to "keep in step" with the changing data distribution.

10. You can't keep your fingers off the finished product:

- OK, so you got 1 to 9 under control, and then put your "holy grail" onto a live account. The bot enters a trade, and it is making money so you just HAVE to close it at a profit, thus ignoring a sound tenet "Let your winners run", and losing 50% of the profit you would have otherwise made (opportunity cost). The bot then enters a trade, and it is losing money, so you "Cut your losing trades short". The second you close your trade, the price action reverses and you lose what would have been a healthy profit. YOU MUPPET!

Now that really ticks me off (Why testing with tick data can be 99% wrong!)

Dontcha just hate it when you spend the weekend back-testing your latest software, only to get disappointing results when you forward test the next week. Where did I go wrong? How can this be happening?

One often mooted reason is the data used for the back-test. If the data is not representative of the live environment, our test results are not likely to be reflecting anything near reality.

Of course, there is the argument that back-testing is intrinsically useless because the future does not reflect the past. Well, a brazillion (large number) folks would disagree. I include among those any trader that uses a support or resistance point, Fibonacci projection or similar in their strategy. What else do these demonstrate but market memory? I digress ...

The list of problems with data includes:

Difference between demo and live: Even if you recorded tick data yourself from a demo server, it may not reflect what is happening on the same broker's live server. Big and frequent differences point to the need to look for a new broker.

Data not reflecting "market" liquidity provider sentiment: Foreign Exchange is a huge market. Retail brokers only have a (usually small) slice. Their quotes will reflect that small share, plus their own spreads to reflect their trading position (after all the majority are actively trading against you). I don't want to reiterate the list of broker "dirty tricks" in this article. Dump a months worth of M1 data, come back a month later and look at the historic data currently shown. You will often find peaks/troughs disappear. These are artificially jacked quotes designed to knockout trade stop losses.

Data gaps: Wow! look at that hugely profitable renko trade! Unfortunately the untradeable stream of bricks produced by my chart generator were entirely due to the gaps in the data I used.

Data granularity: Using 1 minute closes to test a scalping system may not be the optimal method, especially if your system artificially generates ticks based on the Open, High, Low and Close of that bar. Metatrader, for example, uses a technique called fractal interpolation to generate fake ticks from the summary bar data.

Data obtained from a different broker: It goes without saying that testing with data from a different broker has its problems, results depending entirely on the correlation between them and prices from the target broker.

Applications requiring volume as a key parameter: Some providers dont provide volume at all with their data (e.g. Gain Capital). Some provide bid and ask volume separately (e.g. Dukascopy). It is questionable in many cases whether the volume numbers are representative of anything.

It's all in the percentages

Bwa ha ha ha! Sorry, let me pick myself up from the floor and dust myself off. There is sooooo much BS talked about 99% vs 25% tick data that I have to expose this right away. I refer to the test summary report MT4 produces when doing back-testing. If you use fractal interpolation, the percentage will be low, like 20 to 25%, right? But if you have tested with good quality data, the percentage will be high, like 99%, right? WRONG! The 25% figure is a number that the strategy tester in MT4 writes into the tick data file. The 99% figure is a number that the PROGRAM that generated the data writes into the tick data file. Simple as that. In version 406 of MT4, it is a double value written to  bytes 232 to 239 of the file. Any reprobate could fake the "quality" of data used in their testing using a very simple program.

Warning note: MT4 tick data DOES change sometimes from build to build. Please ensure you are using tick data generated for your build, especially if you purchase it.

Solution

There are several solutions to the issues mentioned above, and whichever you take will depend on the type of testing you want to do, whether you are scalping or working on longer time-frames, using time-based or price-based charts.

Some very nice scripts are freely available for downloading, and given time, one can become very proficient at using them. One problem for me is that they usually run very slowly, very frustrating for me if I want to generate a tick data set for several currencies and time-frames on the fly. Being time-conscious, I had to bite the bullet and spend some time, firstly deciding what approach suited me, and then implementing a system. The beauty of porting to a more modern environment such as VB/Java/C++ allows me, as a programmer, to revisit all of the programmer decisions that must have been made multiple times by the various implementers. e.g. Once you move out of MQL4, you need to implement your own StrToTime function, which is fun! As I mentioned previously, quick access to representative source data is important to me, so using Dukascopy with its wacky interface is largely out of the question. My choice of data provider allows me to download in bulk, on the fly, in a matter of minutes. My new system allows me to quickly convert this into a tick data file with 99% quality (or is it 100%?), (big smile .....). BTW, did I mention it can also generate renko tick data?