120115 - Watcher Update

At the risk of having a margin call that will completely ruin my mood, I have decided to post here my latest Watcher version results.

Watcher is my grid trading robot, built on the Metatrader4 platform. For the last 2 weeks, I have been trading it live with results that match the back-testing.

As of Friday afternoon Watcher is up +42%, and I am out of the market totally until Tuesday morning, or possibly NY session on Monday if I get back early enough from my current trip. The reason for taking a break is uncertain market direction from here. Futures open interest shows large speculators very short, the usual signal that precedes a rally. 

On a side note, my DR backup server is having some problems. I am using TeamViewer7 to control it remotely, and have some issues with logging in after reboots. This trip to deepest, darkest Quezon province has provided an excellent opportunity for me to shake this out.

I have been doing a LOT of reading this weekend, and have some great ideas on how to formulate a method for evaluating grid trading systems. This will allow me to perform significantly quicker testing for any new features. I am very keen on spreadsheet modelling, and this method will allow me to evaluate new techniques before coding.

My live experience is matching exactly with the forecast, though I am only shooting for 50% of forecast. Alpha-adding moves have added about 20% to the PL. On friday, when the fall started, I moved from a 20 pip to a 10 pip grid, and doubled my take profit to 20 pips. Then I just watched it fall.When I was down to the final 2 open trades, I switched Watcher off. As the price reached the trough, my last trade closed, and I was out of the market with +42% for the 2 weeks.

Now I am sure there are bruises to come, but manipulating the parameters on this version is way better than last time i tried this, so I will move to automating the main parameters, possibly running them from global variables again, rather than risk going through the EA inputs and screwing it up. Another thing I have done is to have a script to modify all existing TPs, so I can increase them independently of the EA.

I need also to do some tweaking on the prices at which I place pending orders, making them exact multiples of the grid width, and so avoiding the few situations where I end up with a couple of trades a few pips apart, which is inefficient.

Now that really ticks me off (Why testing with tick data can be 99% wrong!)

Dontcha just hate it when you spend the weekend back-testing your latest software, only to get disappointing results when you forward test the next week. Where did I go wrong? How can this be happening?

One often mooted reason is the data used for the back-test. If the data is not representative of the live environment, our test results are not likely to be reflecting anything near reality.

Of course, there is the argument that back-testing is intrinsically useless because the future does not reflect the past. Well, a brazillion (large number) folks would disagree. I include among those any trader that uses a support or resistance point, Fibonacci projection or similar in their strategy. What else do these demonstrate but market memory? I digress ...

The list of problems with data includes:

Difference between demo and live: Even if you recorded tick data yourself from a demo server, it may not reflect what is happening on the same broker's live server. Big and frequent differences point to the need to look for a new broker.

Data not reflecting "market" liquidity provider sentiment: Foreign Exchange is a huge market. Retail brokers only have a (usually small) slice. Their quotes will reflect that small share, plus their own spreads to reflect their trading position (after all the majority are actively trading against you). I don't want to reiterate the list of broker "dirty tricks" in this article. Dump a months worth of M1 data, come back a month later and look at the historic data currently shown. You will often find peaks/troughs disappear. These are artificially jacked quotes designed to knockout trade stop losses.

Data gaps: Wow! look at that hugely profitable renko trade! Unfortunately the untradeable stream of bricks produced by my chart generator were entirely due to the gaps in the data I used.

Data granularity: Using 1 minute closes to test a scalping system may not be the optimal method, especially if your system artificially generates ticks based on the Open, High, Low and Close of that bar. Metatrader, for example, uses a technique called fractal interpolation to generate fake ticks from the summary bar data.

Data obtained from a different broker: It goes without saying that testing with data from a different broker has its problems, results depending entirely on the correlation between them and prices from the target broker.

Applications requiring volume as a key parameter: Some providers dont provide volume at all with their data (e.g. Gain Capital). Some provide bid and ask volume separately (e.g. Dukascopy). It is questionable in many cases whether the volume numbers are representative of anything.

It's all in the percentages

Bwa ha ha ha! Sorry, let me pick myself up from the floor and dust myself off. There is sooooo much BS talked about 99% vs 25% tick data that I have to expose this right away. I refer to the test summary report MT4 produces when doing back-testing. If you use fractal interpolation, the percentage will be low, like 20 to 25%, right? But if you have tested with good quality data, the percentage will be high, like 99%, right? WRONG! The 25% figure is a number that the strategy tester in MT4 writes into the tick data file. The 99% figure is a number that the PROGRAM that generated the data writes into the tick data file. Simple as that. In version 406 of MT4, it is a double value written to  bytes 232 to 239 of the file. Any reprobate could fake the "quality" of data used in their testing using a very simple program.

Warning note: MT4 tick data DOES change sometimes from build to build. Please ensure you are using tick data generated for your build, especially if you purchase it.

Solution

There are several solutions to the issues mentioned above, and whichever you take will depend on the type of testing you want to do, whether you are scalping or working on longer time-frames, using time-based or price-based charts.

Some very nice scripts are freely available for downloading, and given time, one can become very proficient at using them. One problem for me is that they usually run very slowly, very frustrating for me if I want to generate a tick data set for several currencies and time-frames on the fly. Being time-conscious, I had to bite the bullet and spend some time, firstly deciding what approach suited me, and then implementing a system. The beauty of porting to a more modern environment such as VB/Java/C++ allows me, as a programmer, to revisit all of the programmer decisions that must have been made multiple times by the various implementers. e.g. Once you move out of MQL4, you need to implement your own StrToTime function, which is fun! As I mentioned previously, quick access to representative source data is important to me, so using Dukascopy with its wacky interface is largely out of the question. My choice of data provider allows me to download in bulk, on the fly, in a matter of minutes. My new system allows me to quickly convert this into a tick data file with 99% quality (or is it 100%?), (big smile .....). BTW, did I mention it can also generate renko tick data?