We’re using cookies, but you can turn them off in Privacy Settings. If you use the site without changing settings, you are agreeing to our use of cookies. Learn more in our Privacy Policy.

2021 Curriculum CFA Program Level II Quantitative Methods

Backtesting & Simulation

Download the full reading (PDF)

Available to members

Introduction

This reading provides an overview of backtesting and simulation of investment strategies. Backtesting and related techniques enable investment practitioners to simulate the performance of investment strategies (especially quantitative strategies) using historical data or data derived from the distributions of historical data, to generate test results, and to analyze risk and return, without investing any real capital in the strategies.

The rise of big data and the increase in computing power have spurred the development and spread of quantitative investing. Almost every major data vendor has available tools that make systematic backtesting and simulation increasingly accessible. Off-the-shelf software allows backtesting and simulation of endless combinations of possible investment strategies, formulation of multifactor models, and construction of investable portfolios. Developing quantitative investment strategies may appear relatively straightforward, but in reality, it is not. However, understanding the steps and procedures, the implicit assumptions, the pitfalls, and the interpretation of results in backtesting and simulation is a prerequisite for proper utilization of these tools and successful development and implementation of investment strategies.

In a CFA Institute survey of nearly 250 analysts, portfolio managers, and private wealth managers on quantitative investment techniques, 50% of respondents reported that they had conducted backtesting of an investment strategy within the past 12 months of the survey date. This result underscores the importance of backtesting (and other simulation techniques) for investors in practice, and this reading is a starting point on the journey to building this core professional competency.

Learning Outcomes

The member should be able to:

  1. describe objectives in backtesting an investment strategy;

  2. describe and contrast steps and procedures in backtesting an investment strategy;

  3. interpret metrics and visuals reported in a backtest of an investment strategy;

  4. identify problems in a backtest of an investment strategy;

  5. describe different ways to construct multifactor models;

  6. compare methods of modeling randomness;

  7. evaluate and interpret a scenario analysis;

  8. contrast Monte Carlo and historical simulation;

  9. explain inputs and decisions in simulation and interpret a simulation; and

  10. demonstrate the use of sensitivity analysis.

Summary

In this reading, we have discussed how to perform rolling window backtesting—a widely used technique in the investment industry. Next, we described how to use scenario analysis and simulation along with sensitivity analysis to supplement backtesting, so investors can better account for the randomness in data that may not be fully captured by backtesting.

  • The main objective of backtesting is to understand the risk–return trade-off of an investment strategy, by approximating the real-life investment process.

  • The basic steps in a rolling window backtesting include specifying the investment hypothesis and goals, determining the rules and processes behind an investment strategy, forming an investment portfolio according to the rules, rebalancing the portfolio periodically, and computing the performance and risk profiles of the strategy.

  • In the rolling window backtesting methodology, researchers use a rolling window (or walk-forward) framework, fit/calibrate factors or trade signals based on the rolling window, rebalance the portfolio periodically, and then track the performance over time. Thus, rolling window backtesting is a proxy for actual investing.

  • There are two commonly used approaches in backtesting—long/short hedged portfolio and Spearman rank IC. The two approaches often give similar results, but results can be quite different at times. Choosing the right approach depends on the model building and portfolio construction process.

  • In assessing backtesting results, in addition to traditional performance measurements (e.g., Sharpe ratio, maximum drawdown), analysts need to take into account data coverage, return distribution, factor efficacy, factor turnover, and decay.

  • There are several behavioral issues in backtesting to which analysts need to pay particular attention, including survivorship bias and look-ahead bias.

  • Risk parity is a popular portfolio construction technique that takes into account the volatility of each factor (or asset) and the correlations of returns between all factors (or assets) to be combined in the portfolio. The objective is for each factor (or asset) to make an equal (hence “parity”) risk contribution to the overall or targeted risk of the portfolio.

  • Asset (and factor) returns are often negatively skewed and exhibit excess kurtosis (fat tails) and tail dependence compared with normal distribution. As a result, standard rolling window backtesting may not be able to fully account for the randomness in asset returns, particularly on downside risk.

  • Financial data often face structural breaks. Scenario analysis can help investors understand the performance of an investment strategy in different structural regimes.

  • Historical simulation is relatively straightforward to perform but shares pros and cons similar to those of rolling window backtesting. For example, a key assumption these methods share is that the distribution pattern from the historical data is sufficient to represent the uncertainty in the future. Bootstrapping (or random draws with replacement) is often used in historical simulation.

  • Monte Carlo simulation is a more sophisticated technique than historical simulation is. In Monte Carlo simulation, the most important decision is the choice of functional form of the statistical distribution of decision variables/return drivers. Multivariate normal distribution is often used in investment research, owing to its simplicity. However, a multivariate normal distribution cannot account for negative skewness and fat tails observed in factor and asset returns.

  • The Monte Carlo simulation technique makes use of the inverse transformation method—the process of converting a randomly generated uniformly distributed number into a simulated value of a random variable of a desired distribution.

  • Sensitivity analysis, a technique for exploring how a target variable and risk profiles are affected by changes in input variables, can further help investors understand the limitations of conventional Monte Carlo simulation (which typically assumes a multivariate normal distribution as a starting point). A multivariate skewed t-distribution takes into account skewness and kurtosis but requires estimation of more parameters and thus is more likely to suffer from larger estimation errors.