Alex Roslin

Frequently Asked Questions: Backtesting

In Uncategorized on September 1, 2009 at 6:04 pm

Q: How do you come up with your trading setups?

A: I start with detrending the price data. This is a step a lot of trading system developers don’t know about or don’t bother to do. I think it’s one of the most important things I’ve learned about creating trading systems.

Here’s what this means: Almost all markets have a long-term bullish (or in some cases, bearish) bias. In other words, on average, the price has gone up or down. It’s very rare for a market to be flat over many years. The problem that results is this: A trading setup that’s optimized for that data may have major trouble when it encounters market conditions that don’t have the same tilt. Detrending the price data is a way of dealing with this problem.

The most effective way I’ve found to do this is calculating the average weekly price change over the entire dataset (I use weekly data because the COT reports come out each week), then subtracting that from the actual price change each week. The new data should have a zero average price gain or loss.

I credit David Aronson’s excellent book Evidence-Based Technical Analysis (one of the best books on trading system development I’ve read) with opening my eyes to the importance of this step. (Thanks also to Matt Caruso of the Canadian Society of Technical Analysts and the website CXOAdvisory for bringing this book to my attention.)

Second step: running the Commitments of Traders data and detrended price data through an application developed by a smart and generous reader. This checks millions of possible trading setups and determines the best according to each group of traders – both for their net positioning as a percentage of the total open interest and their total open interest (long plus short positioning) – and various trade delays.

The setup parameters are the same in each case: (1) a moving average/standard deviation period (in weeks), (2) a standard deviation value above and below the moving average, which is my signal line, and (3) a trade delay for execution of the trade after a signal (in weeks). I test moving average/standard deviation periods up to about 38 weeks. This is so as not to use up more than 10 percent of the “degrees of freedom” in the dataset. (The typical COT market dataset between March 1995 and Dec. 2007 has 667 weeks. This means the moving-average period plus the number of trading rules in my strategy multiplied by 10 must not surpass 667 when added to the “overhead” number of weeks before the first signal can occur. See Robert Pardo’s book The Evaluation and Optimization of Trading Systems for more details.)

Third step: bringing the top few hundred setups into an Excel spreadsheet and evaluating them with Student’s T-tests for profitability and beating the market, Robust Sharpe score, a Monte Carlo test involving 6,000 randomized market runs, out-of-sample tests, walk-around tests (checking the results for “neighbouring” setups with slightly varied parameter values) and Monte Carlo tests done on those neighbouring setups.

I discard setups that don’t achieve at least a 70-percent volatility-adjusted out-of-sample efficiency for Compound Annual Growth and Regressed Annual Return. I also discard any that don’t achieve at least a 99-percent Monte Carlo score, a 99-percent confidence interval for profitability on a Student’s T-test and a 98-percent confidence interval for beating the market. I also don’t consider setups that use more than 38 weeks in the moving average/standard deviation period (and a smaller period for markets with less data). That’s because I don’t want to use more than 10 percent of the available degrees of freedom in the dataset.

Fourth step: combining the top setups to see how using two or three signals can improve results.

Q: I found a trading setup that was far more profitable than yours in backtesting. Do you want to see the results?

A: Sure. This is a work in progress, and I certainly might not have found the best setup. However, it’s actually not hard to find setups that gained more in backtesting. The question is, How robust is the setup? That’s pretty important. Robustness means the setup can be traded in real time and gain a similar amount as in backtesting – or at least not too much less (or perhaps even more). Without following the steps I’ve outlined above – or a similar process of checking your setup’s robustness – the setup you found may perform quite differently in actual trading.

Q: How often do you plan to update your backtesting?

A: This system probably should be updated with new data after 19 to 38 months – or an eighth to a quarter of the original dataset. That’s based on the original dataset that in most case covers 667 weeks – i.e. March 1995 to December 2007. I started trading the first of my current group of setups in Dec. 2008. At this point, I’m planning to trade these setups for a year and will probably start updating them early in 2010. If the system was tested properly, my current setups should be among the top ones with the new data.

  1. I am wondering if you can suggest books on statisticss for people who don’t have a good understanding of them but are willing to learn.

    You are a journalist. How did you learn all that you know about statistics?

    • Hi Aly,

      I didn’t know a heck of a lot when I started on this, but a lot of input from readers helped guide me to some useful books. One of most useful in my opinion was David Aronson’s Evidence-Based Technical Analysis. It’s not so much about statistics as trading system development. Also interesting are Robert Pardo’s Evaluation and Optimization of Trading Strategies, Curtis Faith’s Way of the Turtle and the classic of trading strategy Reminiscences of a Stock Operator.


  2. Alex –

    Your S&P setup has been bearish for a month now and the market is up close to 7% since then. I imagine you are getting close to being stopped out. My question for you is, have you seen an instance like this where the signal has been bearish for such a long period of time and yet the market continues to move up? And if so, what has been the outcome? If the Large Commercial Traders have been on the wrong side of the trade, it seems like it could turn into a pretty powerful move up if they changed their stance. On the other hand, if they are just setting up the Small Retail Traders, it could be a pretty powerful move down.

    In light of the new highs today, I’d love to hear any input. I really don’t understand how the market moves like this on a day with the kind of news we had.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: