110304 - Thought Control or How (not) To Create A (losing) Winning System

As a systems designer/developer, I am often asked to validate certain client ideas. Leaving aside all judgement of the proposal for a while, I set out first to ensure I thoroughly understand it. Then, using my own experience, I think of all the possible reasons why the hypothesis is flawed. I treat all proposals as mission critical systems, which they are, since our hard-earned cash is at stake.

Should my negative-thinking be proved wrong, or the thoughts inconclusive, I then set up some experiments to assess the validity of the idea.

For automated trading, here be dragons. We code up a rudimentary prototype, with all the features hypothesized to give the design a successful outcome. We back-test, optimize, and then look at the results. If we can manage a decent-looking equity curve, our hypothesis is confirmed!

Or is it? Did we use typical data? Is our optimization (in terms of operating parameters) valid? Did we stumble across the one golden trade to the exclusion of others which gave us a huge balance? Did we need to make 1000 trades in order to get $100 profit? Was the account draw-down exhibited in our test results enough to make us think again? The fact is that the test results rely on the characteristics of the test data. If it is unfeasible that those characteristics will re-occur, then it is unlikely we will achieve the same profit.

Unfortunately, most of our critical thinking in this respect is clouded by what psychologists call Confirmation Bias. This, put simply, is our desire to believe arguments which reinforce our own. Instead of asking questions, or carrying out experiments, which would disprove our hypothesis, we seek out only those that reinforce it.

On the other hand, listening to opposing views and not dismissing unpalatable evidence may be the key to finding what we are looking for. By rejecting the negatives for a sound reason other than they don't confirm our own beliefs, we may be on the right path. By using the negatives to change our own thinking, we may save ourselves time and financial loss.

I love to read through the forums, and read them in the same style that I read my email, newest to oldest unread. In that way, I get the most recent, hottest information first, which allows me to take into context the (time) prior data, and evaluate it in another way. We have nearly 10 years of forum data online, and an amazing wealth of knowledge on tap, in terms of what real developers thought, how they acted, and what the results were.

I find the process of discovery and idea validation the most exciting aspects of my job, and so also provide consultancy along this line. Even if the client does not accept my proposal, or wishes to choose a more sympathetic vendor, I feel obliged to share with them my process and results. Should my own thinking be contradicted, I can always, at my own expense, revisit the counter-argument and reevaluate my position.

Learn like you will live forever. Live like you will die tomorrow, because one day you will be right ...

Further reading on Confirmation Bias