Backtesting & Simulation
Refresher reading access
Introduction
This reading provides an overview of four techniques used to evaluate investment strategies. The first technique, known as backtesting, tests a strategy in a historical environment, usually over long periods, answering the question “How would this strategy have performed if it were implemented in the past?” The second technique, historical scenario analysis, also known as historical stress testing, examines the efficacy of a strategy in discrete historical environments, such as during recessions or periods of high inflation. The third technique, simulation, explores how a strategy would perform in a hypothetical environment specified by the user, rather than a historical setting; it is a useful complement to other methods because the past may not recur and only a limited number of all possible future observations for important variables (e.g., interest rates, return correlations, economic growth) is represented in history. Finally, we explore sensitivity analysis, which is often combined with simulation to uncover the impact of changing key assumptions.
Increasingly powerful off-the-shelf software has moved these techniques from the realm of specialists to generalists. In a CFA Institute survey of nearly 250 analysts, portfolio managers, and private wealth managers, 50% of respondents reported that they had performed backtesting analysis on an investment strategy in the past 12 months. Although performing these analyses now has fewer technical challenges than before, understanding the steps and procedures, the implicit assumptions, the pitfalls, and the interpretation of results have only increased in importance given the proliferation of these tools. This reading is a starting point on the journey to building this core professional competency.
Learning Outcomes
The candidate should be able to:
- describe objectives in backtesting an investment strategy
- describe and contrast steps and procedures in backtesting an investment strategy
- interpret metrics and visuals reported in a backtest of an investment strategy
- identify problems in a backtest of an investment strategy
- evaluate and interpret a historical scenario analysis contrast Monte Carlo and historical simulation approaches
- explain inputs and decisions in simulation and interpret a simulation; and
- demonstrate the use of sensitivity analysis
Summary
In this reading, we discuss how to perform rolling-window backtesting—a widely used technique in the investment industry. We also described how to use scenario analysis and simulation along with sensitivity analysis to supplement backtesting, so investors can better account for the randomness in data that may not be fully captured by backtesting.
- The main objective of backtesting is to understand the risk–return trade-off of an investment strategy by approximating the real-life investment process.
- The basic steps in rolling-window backtesting are specifying the investment hypothesis and goals, determining the rules and processes behind an investment strategy, forming an investment portfolio according to those rules, rebalancing the portfolio periodically, and computing the performance and risk profiles of the strategy.
- In the rolling-window backtesting methodology, researchers use a rolling-window (or walk-forward) framework, fit/calibrate factors or trade signals based on the rolling window, rebalance the portfolio periodically, and then track the performance over time. Thus, rolling-window backtesting is a proxy for actual investing.
- Analysts need to pay attention to several behavioral issues in backtesting, including survivorship bias and look-ahead bias.
- Asset (and factor) returns are often negatively skewed and exhibit excess kurtosis (fat tails) and tail dependence compared with a normal distribution. As a result, standard rolling-window backtesting may be unable to fully account for the randomness in asset returns, particularly on downside risk. ■ Financial data often face structural breaks. Scenario analysis can help investors understand the performance of an investment strategy in different structural regimes.
- Historical simulation is relatively straightforward to perform but shares pros and cons similar to those of rolling-window backtesting. For example, a key assumption these methods share is that the distribution pattern from the historical data is sufficient to represent the uncertainty in the future. Bootstrapping (or random draws with replacement) is often used in historical simulation.
- Monte Carlo simulation is a more sophisticated technique than historical simulation. In Monte Carlo simulation, the most important decision is the choice of functional form of the statistical distribution of decision variables/ return drivers. Multivariate normal distribution is often used in investment research, owing to its simplicity. However, a multivariate normal distribution cannot account for negative skewness and fat tails observed in factor and asset returns.
- Sensitivity analysis, a technique for exploring how a target variable and risk profiles are affected by changes in input variables, can further help investors understand the limitations of conventional Monte Carlo simulation (which typically assumes a multivariate normal distribution as a starting point). A multivariate skewed t-distribution considers skewness and kurtosis but requires estimation of more parameters and thus is more likely to suffer from larger estimation errors.
2 PL Credit
If you are a CFA Institute member don’t forget to record Professional Learning (PL) credit from reading this article.