Get 40% Off
🤯 This Tech Portfolio is up 29% YTD! Join Now to Get April’s Top PicksGet The Picks – Just 99 USD

A Caveat On Backtesting Caveats

Published 01/13/2017, 07:48 AM
Updated 07/09/2023, 06:31 AM

Ben Carlson at Ritholtz Asset Management reminds us that backtesting offers no shortcuts to investment nirvana. As he correctly points out, there are numerous shortcomings in the art/science of reconstructing the historical results of an investment strategy. But it’s also true that backtesting, if used wisely, can be a powerful tool for sensibly managing expectations with regards to return and risk. In fact, there’s really no reason not to run a backtest, even on strategies that appear to be free of mystery. The tough part is figuring out how, or if, to use the results. Fortunately, clear thinking and planning can boost the odds that backtesting will be a productive exercise.

Even under the best of circumstances, however, it’s essential to remember that backtests are but one tool for researching portfolio design. And like most tools in finance, this one has its share of traps. Meantime, some of the most valuable information in a backtest may counterintuitive because it’s bound up with negative selection. An intelligently designed backtest that shows a strategy to be a dog is usually far more persuasive than a backtest that offers a glowing review.

At its core, this is a tool for data exploration and studying how a given set of portfolio rules interact with securities. But the devil’s firmly in control of the details and you can’t spend too much time thinking about best practices in this corner of quantitative analysis. The first order of business, as Brian Peterson advises in his must-read “Developing & Backtesting Systematic Trading Strategies,” is laying out priorities before you write one line of code.

It is important to understand what you are trying to achieve before you set out to achieve it.

Without a set of clearly defined goals there is great potential to accept a strategy or backtest that is really incompatible with the (previously unstated) business objectives and constraints. Worse, it can lead to adjusting your goal to try to follow the backtest, which can culminate in all sorts of bad decision making, and also increases the probability of erroneously accepting an overfitted backtest.

The sad reality is that backtesting is abused, sometimes naively but intentionally as well. The worst offenders are the charlatans who use flimsy backtests to hawk questionable investment strategies and products.

The good news is that an well-constructed backtest is enormously valuable, if only because it forces you to analyze your assumptions by writing and review the rules via computer code. Nearly every coding project I’ve worked on delivered at least one “ah-ha!” moment of deeper clarity about the pros—and cons—of a strategy under scrutiny. In fact, every strategy deserves to be backtested so that the investor has a clearer understanding of the mechanics. Having coded countless strategies over the years (mostly in R), I’ve found the work to be profoundly enlightening, even when the results more or less fall in line with expectations.

There’s an assumption that only complicated, short-term trading strategies are fair game for backtesting. But relatively conservative portfolio strategies with medium- and long-term horizons can benefit from backtesting analysis by shedding light on the inner workings of interactions between, say, a dozen ETFs that are rebalanced periodically. You may or may not adjust the strategy after this analysis, but the process of quantifying assumptions and reviewing how the risk and return metrics stack up is almost always an edifying effort. The insight may be subtle—recognizing that a fund offers far less diversification benefits than assumed, for instance. Or maybe the results reaffirm your expectations. Another possibility: a backtest reveals a previously overlooked weakness in the strategy.

Backtesting as an educational tool, in other words, is arguably where the most value-added insight lies. By contrast, hitting a home run by uncovering a previously unknown investing system that generates high returns with low risk is the equivalent of winning the lottery—it’s possible, but unlikely.

The simple truth is that there’s no avoiding backtests. Even a buy-and-hold strategy is based on a backtest, albeit a simple one.

We shouldn’t whitewash the caveats in backtesting, but that’s no reason to assume the worst. The potential for abuse is vast, but so is the opportunity to inform and illuminate.

Latest comments

Risk Disclosure: Trading in financial instruments and/or cryptocurrencies involves high risks including the risk of losing some, or all, of your investment amount, and may not be suitable for all investors. Prices of cryptocurrencies are extremely volatile and may be affected by external factors such as financial, regulatory or political events. Trading on margin increases the financial risks.
Before deciding to trade in financial instrument or cryptocurrencies you should be fully informed of the risks and costs associated with trading the financial markets, carefully consider your investment objectives, level of experience, and risk appetite, and seek professional advice where needed.
Fusion Media would like to remind you that the data contained in this website is not necessarily real-time nor accurate. The data and prices on the website are not necessarily provided by any market or exchange, but may be provided by market makers, and so prices may not be accurate and may differ from the actual price at any given market, meaning prices are indicative and not appropriate for trading purposes. Fusion Media and any provider of the data contained in this website will not accept liability for any loss or damage as a result of your trading, or your reliance on the information contained within this website.
It is prohibited to use, store, reproduce, display, modify, transmit or distribute the data contained in this website without the explicit prior written permission of Fusion Media and/or the data provider. All intellectual property rights are reserved by the providers and/or the exchange providing the data contained in this website.
Fusion Media may be compensated by the advertisers that appear on the website, based on your interaction with the advertisements or advertisers.
© 2007-2024 - Fusion Media Limited. All Rights Reserved.