Ever hear one of these criticisms when the subject of stock market back-testing is discussed?
- “But those are just back-tested results and not what is going to happen.”
- “Past performance doesn’t guarantee future results.”
- “Anyone can look in the past after the fact and realize that Apple (AAPL) was a great stock but that doesn’t help you strategically pick the next winner.”
As someone who builds portfolio strategies I hear these sorts of arguments almost every week. On the surface it seems to make sense – that living in the past will not tell you how to invest for the future. It might stand to reason that buying shares with a firm still making the Atari 2600 or war catapults is unlikely to reap big rewards going forward. On the other hand, the book of history has been carefully preserved for us to learn from. If we do not examine the past, might we be ignoring valuable lessons that were taught at great cost to our forefathers?
To discover the truth about basing future decisions on past results we will attempt to back-test and build a forecasting model that predicts something other than stocks… the likelihood of war.
Back-Testing And Forecasting War
In the following example we will have three very different men trying to determine when the next war might take place. Each of them has access to an unlimited amount of historical data. The year is 1950.
The Technical Analyst
The first man is considered to be a technical analyst. He is a time-series numbers guy that is good at those brain teasers where you need to determine what the next number is in the sequence (1,3,5,7,?). The technical analyst pulls out a timeline and looks at previous wars. After examining the timeline, he announces the creation of a formula to determine when the next major wars will occur. His chart is below.
After examining his chart for a few minutes (remember we are in 1950) you may realize what his forecasting time-series indicator really is. The gap between when Hitler was born and 1914 is 25 years. The time elapsed between the start of WWI and WWII is 25 years. He therefore concludes that every 25 years a major world event related to war will occur.
Is this sound reasoning or is he trying too hard to find correlations in random data?
The Fundamental Analyst
The second person fancies himself as a fundamental analyst. His focus is not on when the wars took place and the years between conflicts, but he focuses instead on country-specific data to determine if a country is preparing for battle. He examines the following figures to determine if there is a correlation between any one factor (or a combination of factors) and when a war erupted:
- Nations with an active military
- Size of military
- Sentiments between countries
- Trade sanctions
- The forecasts of other war analysts
- Economic conditions
It may very well be that he discovers a link between countries that have trade sanctions with big military forces and negative feelings toward each other combined with newspaper forecasts of imminent war with a subsequent occurrence of actual armed conflict. The fundamental analyst might monitor these same factors to determine when the threat of war is more probable in a given area.
The Strategist
The third man is a strategist. He is not content with only uncovering the factors that correlate to war – he wants to understand what led up to the battle. The strategist will likely use all of the same indications as the fundamental analyst but he takes it one step further. He is obsessed with understanding why the war began.
The strategist cannot always clearly define his process. He may sift through reams of data trying to "get a feel" for the bigger picture. He might take long walks on the beach meditating and mentally digesting what he has learned. Intuition (leaps of logic he cannot fully explain) may also play a part. Unexpectedly, in the middle of the night, an epiphany strikes and he begins to assemble all the pieces necessary for a theory. He writes down a few lines of logic to develop and later test with rigor:
- Oppressed nations are more likely to revolt
- Nations attempting to develop nuclear weapons may incite war from countries that already have the technology and want to protect it or nearby nations that are afraid of domination
- A powerful nation in an economic slump might go to war in the hopes of it boosting the economy
- A poorly protected nation may become a target if they have much needed resources that are globally scarce such as water, oil, lumber or gold
The strategist may concoct all sorts of other theories as to what the real cause of any given war may be. But he does not "trust his gut." He then tests these theories in a variety of ways (which will be discussed in part 2 of this article). If his theories hold up, he then isolates any country and nation that meets the above criteria and puts them on his "danger short-list." He also monitors for arms build-up, sentiment and other factors if these have been somewhat reliable indicators in the past. If a war breaks out that doesn’t fit into his theory, he analyzes the cause and includes this new data with his old strategy.
Everyone Is A Back-Tester But Doesn’t Know It
In the example above, are all three analysts using historical data to forecast a future event to some degree? Absolutely. Should you trust all three analysts equally since all are back-testers? Absolutely not. This aptly illustrates the variability of stock market practitioners who use strategy back-testing and the reason why they should not all be painted with the same brush.
In a sense, anything that has happened in the past, and that you think might happen again, is considered back-testing. Did you read that people fall when jumping off a cliff (with a regular suit as depicted here and not a wing-suit)? Or that drinking and driving causes more fatal accidents? This is all typically based off historical back-testing. Theorizing after the event should not necessarily discount the theory and you don’t always need to test the results first-hand before believing it. Instead, think of it as collecting and examining the evidence to develop theories that need to be tested with scrutiny.
If you can explain what, how and when… this is good. If you understand why it may have happened or what the causal agents are, you have discovered something better than gold.
Back-Testing Answers Questions
Here are some of the questions relating to the stock market that is difficult to answer without historical data and therefore back-testing:
- Is owning equity shares a good investment?
- What are some warning signs of a stock about to cut its dividend?
- Is it profitable to re-invest dividends in the same company or look for another stock that seems good?
- Is it better to buy a great company with over-valued shares or a so-so company with cheap shares?
- Are small companies riskier than large companies? Do they typically offer better returns?
- Is there a relationship between profit and share price?
And the list goes on and on. What I wonder is this: if you ignore everything that has happened in the past, how do you decide what a good investment is?
The truth of the matter is that many people who condemn back-testing are the worst type of back-testing analysts without even knowing it. They may do some of the following since they are told these techniques and methods work – but they likely have not tested it for themselves:
- Selling when stocks fall below the 200 day moving average
- Selling in the middle of a bear market
- Buying after prices shoot up
- Using stop-loss orders
- Buying stocks at 52-week lows
- Buying companies getting the most media coverage
- Believing that diversification lowers risk of portfolio loss
- Thinking that a volatile stock is always more risky than a slower moving stock
The real limitations of back-testing comes, not from looking at historical data and making assumptions, but in how this process is carried out and by whom. Will a war occur every 25 years on the dot? Probably not. Will you fall if you jump off a bridge? More than likely and you shouldn’t test that out either.
If you look at historical data with a critical eye and sound logic, you might discover what causes some stocks to rise and others to fall. It is possible that, in time, you might even be able to build a strategy that takes advantage of a certain situation for a little extra gain.
On the other hand, if you are simply running some technical indicator over the price data in the hopes of finding the next Apple or Microsoft, you will likely suffer bitter disappointment when your portfolio implodes over time despite using some futuristic looking strategy back-testing software that makes pretty charts purporting astronomical returns.