Frequently Asked Questions: How It Works

Q: Your setup for the S&P 500 just went bullish on Monday’s open, and the index is already down 2 percent as of Tuesday. Are you throwing in the towel?

A: No, it doesn’t work that way. First, this is a mechanical system, and that means following the signals with discipline, even when it’s down. Also, if you take a look at my backtesting results for this setup and my other ones, you’ll see they haven’t had a 100-percent success rate. No trader or system does. Even the best traders typically have only a 60-percent win record.

Because of that, I am prepared to lose money on many of my trades. I set a stop level and position size that’s appropriate to my risk level based on past volatility and losses in a setup. (See more on risk-control, stops and position sizing on my How It Works page.)

Also, my signals last an average of three or four weeks, and lots can happen between now and then. And it takes more than a single loss or two to make me question my system, which is based on results from dozens of trades and years of data.

A: Your signal has made money, but the market has turned down. Why are you waiting for your setup to give a new signal before you get out?

Q: Firstly, a trading strategy is useless if you don’t follow its signals with 1,000,000% (is there such a thing?) discipline. All your work developing it goes right into the toilet. A trader needs to have confidence his or her system will work or go back to the drawing board and improve it so they do. Or give up, and do something else. Whenever I start second-guessing ourselves, that’s when the big mistakes and losses start piling up.

I don’t care at all what the price is doing right now. I don’t really even care that much if this or another trade wins or loses money. All systems lose money some of the time. The key is winning over the long-term and cutting losses with stops and position sizing.

In any event, who knows what will happen between now and when the sell signal comes. Maybe the market will go back my way!

Q: What would make you question a losing signal?

A: If a trade loses more than my stop level, I automatically sell my position. If the trade loses more than the setup has ever lost since the data started (usually in 1995 for most Commitments of Traders markets), I will cease trading the setup for four weeks. I call that my Black Swan Rule. Read more about all this at my How It Works page. (In trading, a “Black Swan” is a hard-to-predict, catastrophic event. The idea comes from Nassim Nicholas Taleb and his acclaimed book, The Black Swan: The Impact of the Highly Improbable.)

As well, if my overall portfolio is down more than six percent in any four-week period, I’ll stop trading the entire system for four weeks.

Q: What is your rationale for the trade delays in your setups?

A: The trade delays are based on my backtesting. I test various trade delays for each setup – from zero to eight weeks. I use the ones that give the best results. Why does it work best to delay a trade in some cases? I can’t say for certain, but I believe it’s because it can take time for major positioning changes to work themselves through into the broader markets. It could also have to do with the differing dynamics of mean-reversion in various markets after extreme positioning occurs in a key trader group.

Q: I’ve read that the Commitments of Traders data is obsolete. It no longer works, probably because too many people know about it. Today, for example, the market is caving in, while the commercial traders – the so-called “smart money” – just got extremely bullish on the S&P 500.

A: Contrary to conventional wisdom, the S&P 500 does not usually go up when the commercial traders get more bullish. In fact, the opposite takes place. The commercial hedger net position has a negative, -25-percent correlation with S&P 500 weekly open prices a week after the COT report comes out (between 1995 and Feb. 2010).

That’s right: a negative correlation. So trading alongside the commercials week to week would have lost you piles of money. That actually makes sense, in a way. These guys accumulate positions as markets decline and vice versa. It’s only when they hit extremes of positioning that I think it’s safe to trade alongside them. Various analysts like Larry Williams discovered this long before I did. In my research, I’ve found the most robust signals usually come when the commercial traders agree with at least one other group of traders, like the small traders, who would usually be faded. That’s why I use two  signals in my own S&P 500 setup and only trade when both agree. On top of that, this particular signal tends to work most reliably not right away – but with a delay of about three weeks on average.

So has the COT data slowly become obsolete? I’ve seen this said from time to time. Again, the reality is quite the opposite. Here are correlations in various time periods between the commercial net position and S&P 500 prices a week later:

– March 1995-Nov. 1997: -71%

– Dec. 1997-Dec. 1999: +14%

– Dec. 1999-March 2003: 0%

– March 2003-June 2008: +34%

– March 2003-Feb. 2010: +53%

– June 2008-Feb. 2010: +63%

– March 1995-Feb. 2010: -25%

(These correlations are not the same in other markets.) So clearly, the COT data has changed dramatically over the various periods in how it interacts with market prices. If anything, the commercial hedgers are now trading much more in lockstep with the S&P 500. But the relationship seems to evolve quite often, so using it alone to trade is probably very much a random exercise. This is why I think the data needs to be filtered with some more complex trading system.

Q: Can you recommend an ETF to trade one of your signals?

A: No, afraid not. I advise readers to check the regularly updated list of ETFs at the DVTechTalk.com website of retired RBC analyst Don Vialoux.

Q: Can your signals be traded with an individual company’s stock? For example, could you buy a specific mining stock to trade your gold signal?

A: Not really. Company shares can be influenced by numerous factors that have nothing to do with the market in question. The further the traded security is from the underlying market, the less it will track it well.

Q: Couldn’t you improve your results with technical indicators based on the price action?

A: My backtesting so far hasn’t shown any price-based indicators that improve the results and robustness of my setups. Mind you, I haven’t tested technical indicators comprehensively. However, as David Aronson points out in his excellent book Evidence-Based Technical Analysis, most commonly used technical indicators don’t make money.

Q: Do you ever put on discretionary trades not based on a COT signal?

A: Yes, but not ones that are directly opposite to a COT signal. For example, if my COT signal is short the S&P 500, I wouldn’t go long the S&P 500. But I may go long a different equity index or some other sector or stock. I generally place discretionary trades based on breakouts of Tom DeMark Setup Trend lines on the daily and weekly charts. See this story I did on DeMark’s interesting indicators to learn more.

Q: Why don’t you have a setup for the e-mini S&P 500 contract or the NASDAQ 100 mini contract?

A: So much to do, so little time. It’s on my to-do list to test lots of other markets, but I just haven’t gotten to it yet. That said, past research of mine suggests trading setups based on the mini contracts haven’t been as robust or given as good results as those for the larger contracts.

Q: Are the commercial hedgers always the “smart money”? Are the large speculators and small traders always the “dumb money”?

A: No to both questions. In most cases, the commercial traders are positioned correctly – i.e. highly long – when a market is about to go up. Conversely for the large specs and small traders, they usually tend to be positioned wrong – i.e. highly short – when a market is about to go up. But this popular understanding of the COT data isn’t always confirmed in backtesting. My testing of the data has shown that those patterns aren’t true of every market. Some of the best signals trade on the same side as the small traders or large specs and fade the commercials. As well, much depends on the timeframe (i.e. the setup parameter values) you use to look at the data. Under some conditions, it might be best to fade the small traders in a market, while in others it’s best to trade alongside them.

Think back again to how the small traders are positioned at a market bottom, for example. As trend followers, they were actually correctly positioned if they were highly short. If they trade with success, a set of parameter values that captures their trend-following positioning would work well.

Q: You have a setup based on the commercial and small trader net positioning, but the large speculators are saying something different this week. For example, they are signaling it’s best to be long, while your setup says to be short. Will you be ignoring your signal?

A: No. My setups are based on backtesting all groups of traders – both for their net positioning and total open interest. The two or three signals I’ve decided to use were the best ones I found. It’s irrelevant what other trader groups are doing, until it’s time to update my backtesting.

Q: Isn’t open interest moving from the large S&P 500 contract to the e-mini contract? Doesn’t that mean your setup based on the larger contract is obsolete?

A: It’s true there’s been steady long-term growth in the commercial total open interest in the mini S&P 500 contract. That actually started pretty much at the beginning of the data in 1997. That, however, hasn’t been reflected in a long-term decline in the commercial total open interest in the larger S&P 500 contract. Neither has it been reflected in any kind of deterioration of trading setup backtesting results in more recent years. The setup I’m using for the S&P 500 has results in the 2003-2008 period that match those in the 1995-2003 timeframe.

It’s also true that the commercial total open interest in the larger S&P 500 contract has fallen a lot since late 2008 (as of this writing, in Aug. 2009). But so has the mini commercial total open interest. As of Aug. 2009, it was still well off where it was last year.

It’s also important when backtesting trading systems to look at more than just one year’s worth of data. Doing that, you’ll notice that the commercial total open interest in the larger SPX contract doubled between June 2008 and its peak last Dec. 2008. So even though it’s true it’s fallen since then, it fell from bubble highs.

A more fundamental question is this: Is the commercial total open interest even a reliable indicator for trading? My research suggests that data is probably not as useful from a trading viewpoint as the signals I did choose to use for my S&P 500 setup – the commercial and small trader net positioning as a percentage of the total open interest. I didn’t get a single potential trading setup out of the total commercial open interest during my backtesting of the S&P 500 COT data.

Q: Your setup based on the commercial traders in the S&P 500 was long while the commercials in the e-mini S&P 500 contract were highly short. Doesn’t that mean your setup was wrong?

A: The commercial traders in the e-mini contract are not necessarily the same folks as those in the larger SPX contract, which is five times bigger. The CFTC’s reporting levels for both contracts are the same. That means the commercial category in the mini contract includes a number of traders who would be considered small traders (i.e. the non-reportable category) in the larger contract.

DISCLAIMER

I created this blog to record my personal, subjective observations about the markets, solely for my entertainment and that of interested readers and to foster research. Nothing on this blog should be construed as financial advice or an offer or recommendation to purchase or sell any security. I encourage anyone interested in the markets to do your own homework and/or consult a professional advisor. I am not a certified financial advisor and am still refining my trading system. I reserve the right to modify or stop publishing aspects of my system at any time. My system has resulted in very large drawdowns in some trades. Past backtested and real-time results are no guarantee of future performance. I will not be liable for any losses or damages of any kind that result from the content of this site.

Although I consider the data, calculations and information on this blog to be reliable, I can’t make any guarantees and won’t be held liable or responsible for anything erroneous on this website. You are solely responsible for implementing safeguards of your data and system when you use this site and its content and links. It is up to the user of this site and content to protect yourself from worms, trojan horses, viruses and the like. I may hold positions in some of the securities mentioned in this blog.

Frequently Asked Questions: Backtesting

Q: How do you come up with your trading setups?

A: I start with detrending the price data. This is a step a lot of trading system developers don’t know about or don’t bother to do. I think it’s one of the most important things I’ve learned about creating trading systems.

Here’s what this means: Almost all markets have a long-term bullish (or in some cases, bearish) bias. In other words, on average, the price has gone up or down. It’s very rare for a market to be flat over many years. The problem that results is this: A trading setup that’s optimized for that data may have major trouble when it encounters market conditions that don’t have the same tilt. Detrending the price data is a way of dealing with this problem.

The most effective way I’ve found to do this is calculating the average weekly price change over the entire dataset (I use weekly data because the COT reports come out each week), then subtracting that from the actual price change each week. The new data should have a zero average price gain or loss.

I credit David Aronson’s excellent book Evidence-Based Technical Analysis (one of the best books on trading system development I’ve read) with opening my eyes to the importance of this step. (Thanks also to Matt Caruso of the Canadian Society of Technical Analysts and the website CXOAdvisory for bringing this book to my attention.)

Second step: running the Commitments of Traders data and detrended price data through an application developed by a smart and generous reader. This checks millions of possible trading setups and determines the best according to each group of traders – both for their net positioning as a percentage of the total open interest and their total open interest (long plus short positioning) – and various trade delays.

The setup parameters are the same in each case: (1) a moving average/standard deviation period (in weeks), (2) a standard deviation value above and below the moving average, which is my signal line, and (3) a trade delay for execution of the trade after a signal (in weeks). I test moving average/standard deviation periods up to about 38 weeks. This is so as not to use up more than 10 percent of the “degrees of freedom” in the dataset. (The typical COT market dataset between March 1995 and Dec. 2007 has 667 weeks. This means the moving-average period plus the number of trading rules in my strategy multiplied by 10 must not surpass 667 when added to the “overhead” number of weeks before the first signal can occur. See Robert Pardo’s book The Evaluation and Optimization of Trading Systems for more details.)

Third step: bringing the top few hundred setups into an Excel spreadsheet and evaluating them with Student’s T-tests for profitability and beating the market, Robust Sharpe score, a Monte Carlo test involving 6,000 randomized market runs, out-of-sample tests, walk-around tests (checking the results for “neighbouring” setups with slightly varied parameter values) and Monte Carlo tests done on those neighbouring setups.

I discard setups that don’t achieve at least a 70-percent volatility-adjusted out-of-sample efficiency for Compound Annual Growth and Regressed Annual Return. I also discard any that don’t achieve at least a 99-percent Monte Carlo score, a 99-percent confidence interval for profitability on a Student’s T-test and a 98-percent confidence interval for beating the market. I also don’t consider setups that use more than 38 weeks in the moving average/standard deviation period (and a smaller period for markets with less data). That’s because I don’t want to use more than 10 percent of the available degrees of freedom in the dataset.

Fourth step: combining the top setups to see how using two or three signals can improve results.

Q: I found a trading setup that was far more profitable than yours in backtesting. Do you want to see the results?

A: Sure. This is a work in progress, and I certainly might not have found the best setup. However, it’s actually not hard to find setups that gained more in backtesting. The question is, How robust is the setup? That’s pretty important. Robustness means the setup can be traded in real time and gain a similar amount as in backtesting – or at least not too much less (or perhaps even more). Without following the steps I’ve outlined above – or a similar process of checking your setup’s robustness – the setup you found may perform quite differently in actual trading.

Q: How often do you plan to update your backtesting?

A: This system probably should be updated with new data after 19 to 38 months – or an eighth to a quarter of the original dataset. That’s based on the original dataset that in most case covers 667 weeks – i.e. March 1995 to December 2007. I started trading the first of my current group of setups in Dec. 2008. At this point, I’m planning to trade these setups for a year and will probably start updating them early in 2010. If the system was tested properly, my current setups should be among the top ones with the new data.