Why Historical Market Data Isn’t as Reliable as You Think
Long-term stock market charts often depict a steady upward trajectory, reinforcing investor confidence in future returns. However, this narrative relies heavily on reconstructed data, survivorship bias, and evolving methodologies that challenge the accuracy of historical performance claims.
Key Challenges in Historical Data Compilation
1. Survivorship Bias: Early market records often excluded failed companies or entire markets. For example, pre-1934, bankruptcies weren’t systematically reported, skewing data toward “winners.” Researchers like Dimson, Marsh, and Staunton (DMS) attempt to correct this by including defunct markets (e.g., pre-Soviet Russia) and scouring archives, but gaps remain.
2. Backtested Hypotheticals: Major indices like MSCI’s World Index (launched 1986) use backtested data for pre-1986 periods, which the firm explicitly labels as hypothetical. Similarly, the S&P 500’s pre-1976 returns are theoretical, as no investable index fund existed.
3. Methodological Shifts: Index construction rules have evolved. Free-float weighting (ignoring closely held shares) and sector classifications (standardised only in 1999) mean historical comparisons are often apples-to-oranges. For instance, 19th-century “market returns” focused on railroads, while today’s indices are tech-dominated.
Actionable Insights for Investors
- Stress-Test Return Assumptions: Avoid relying on century-long averages. Use tools like compound interest calculators with conservative return ranges (e.g., 4-7% real returns for equities).
- Diversify Geographically: The US’s 20th-century outperformance is clear in hindsight but wasn’t predictable in 1900. Global diversification mitigates survivorship bias risks.
- Focus on Post-1990 Data: For global markets, reliable real-time data begins in the 1990s. Pre-1970s records are patchy, especially outside the US and UK.
The Bottom Line
While markets have historically trended upward, the further back we look, the more data becomes a blend of scholarship and educated guesswork. This doesn’t negate equities’ long-term potential but underscores the need for humility in forecasting. As DMS’s work shows, rigorous analysis helps, but uncertainty remains – plan accordingly.