Sector Rotation

The question

Formally this is a state-estimation problem. The latent state is rotation phase; the observables are noisy sector-level returns and breadth. The deliverable is a calibrated posterior probability over rotation phase at daily frequency, using data through today only — with two kinds of noise removed first: market-wide movement, and the distortion from a single dominant constituent (a cap-weighted sector index is hostage to its largest holding).

Data — owned, not borrowed

The project owns its price feed on one consistent, single-source basis rather than reading a shared, multi-vendor table. That discipline is deliberate — mixing vendors with no cross-source resolution lets one bad print corrupt returns invisibly. Each return is computed within a single source; where two sources meet, they are joined by return, not by splicing incompatible levels.

• Prices — 7.37M daily bars from a delisting-complete vendor, 1990-2026. The companies that left the index are still present, so the dataset is free of survivorship bias — the single most important property for this question, because names that leave the index are systematically the worst performers and cluster in exactly the sectors rotating out.
• Membership & weights — S&P 500 constituents and float weights as they actually stood each quarter, joined by ISIN, never ticker (tickers silently recycle — the same symbol can point at two different companies over time).
• Cross-checks — prices reconcile to a second independent feed at 99.85% agreement; cap-weighted sector returns reconcile to independent sector-ETF benchmarks at 0.984 correlation. Disagreements are flagged, never silently overwritten.

Denoising

Before modeling, each sector return is taken net of its own trailing beta to the market, leaving a market-denoised residual. Alongside it we compute breadth (how many members participate) and concentration (how much of the move is one name). A genuine rotation shows broad participation; a fake one driven by a single mega-cap does not — and the breadth/concentration measures are the direct discriminators.

The mechanical ruler

To detect a rotation you must first define one. The definition is pre-registered and frozen before any model is fit: cross-sectional dispersion above its expanding percentile (ranked against all history to date), for a minimum run of days, with sufficient breadth in the leading sectors and low rank-correlation to the prior ordering. This mechanical rule is an external reference the model never sees — so "state 2 is a rotation" becomes checkable, not asserted.

The model

The reading comes from a discrete hidden Markov model (three states) over the sector observables. Two disciplines matter most:

• Filtered, not smoothed. The probability on each date uses observations through that date only. Smoothed (full-sample) states look dramatically better and are useless in real time; they are never used to score anything. The model's parameters, however, are fitted on 1990–2020 as a whole, so its values before 2021 are in-sample. The walk-forward backtest (below) is the record with no look-ahead at all.
• Calibration. The raw model score is mapped to a probability by isotonic regression fitted on the model's own 1990–2020 output (in-sample). The walk-forward backtest fits the same map only on earlier out-of-sample forecasts.

The model is unsupervised — it never sees the mechanical ruler. That one of its states independently lines up with the ruler is the evidence that the state is real rather than an artifact of fitting.

Validation

Rotations are rare, so a single train/test split cannot confirm anything. The evidence is built in two stages:

• Blocked, purged cross-validation within 1990–2020 — contiguous blocks with a gap at each boundary so nothing leaks through the trailing windows. Median out-of-sample AUC 0.913 (a median over folds, some of which hold few episodes).
• A holdout on 2021+ that the fit never uses. It is re-scored every day as new days arrive, with the 1990–2020 fit frozen: AUC 0.8697, calibration error 0.0054.
• The walk-forward backtest (next section) — the strictest of the three.

Results are reported against the unconditional base rate (14.1%); the walk-forward backtest also compares the model with the mechanical rule's same-day check and with the dispersion percentile alone, so the latent-state machinery has to earn its complexity. The model was fitted on 109 rotation episodes from 1990–2020.

How we backtest — the walk-forward test

The published model is fitted once on 1990–2020, which makes its history before 2021 in-sample. The walk-forward backtest removes that: at the end of every month from December 1995 it refits the same model (the same standardisation, three-state hidden Markov model and settings) on the data available that day only, and scores every day of the following month with that fit. The probability map (calibration) is fitted only on earlier forecasts whose outcome was already known at least five trading days before. Results start on January 2, 2001, once five years of such forecasts exist; 2021 onward is reported separately as the held-out period.

Every question, metric and pass rule was written down before the first run (the pre-registration, September 29, 2026). A result is Supported when it has the direction the page claims, is clear of chance (t ≥ 2, or a 95% range that excludes no effect) and holds in the held-out period; Contradicted when it is clearly the opposite; otherwise Not supported. Ranges come from a block bootstrap (blocks of 63 trading days) or, for the young-rotation flag, from resampling whole runs.

• S1 — Does the probability identify rotation days better than always guessing the average rate? (Brier skill score: 0 = no better, 1 = perfect.) Primary test. Supported.
• S2a — Does it beat the rotation rule's own same-day check (the three conditions met today, before five days confirm)? Supported.
• S2b — Does it beat the dispersion percentile on its own? Supported.
• S4 — After the probability crosses 0.50, is a rotation confirmed within 20 trading days more often than after an average day? Supported.
• S5 — Does the young-rotation flag rank young runs better than a model that only knows how long each run has lasted? Supported.
• S6 — In a rotation state, are last month's sector leaders a worse guide to next month than in a calm state? Not supported.
• S7 — Does a sector-fund strategy that holds the 3 recent leaders, but steps aside to all funds equally when the probability is 0.50 or more, beat always holding the 3 leaders? (costs 5 bps per trade) Not supported.

Two benchmarks are turned into probabilities the same way as the model: the rotation rule's own same-day check (its three conditions met today) and the dispersion percentile alone. The trading test uses the eleven sector ETFs, trades at the close after each month end, and charges 5 and 10 basis points per unit of turnover. The flag test refits the flag's logistic model at every year end on runs whose outcome was already known, with the macro inputs lagged one trading day (their source publishes a day's value on the next business day).

Limits. Each company keeps its current GICS sector for its whole history; the rotation rule's thresholds were chosen on 1990–2020; the 0.50 band was chosen after looking at 2021+ results, so the held-out crossing figures are not a clean test (the live record from 16 July 2026 is); trading costs are fixed estimates. The backtest runs every day with the page, and a change to any rule is a new method version, rebuilt in full.

Is a new rotation starting? — the onset flag

"Are we in a rotation?" and "is a new one just beginning?" are different questions. The state model answers the first. The young-rotation flag answers the second. The frozen definition requires five consecutive qualifying days, so a rule-follower cannot know on day one — on day k<5 the flag forecasts the probability this nascent run reaches five and confirms. The training data was free: every past run that died at one to four days is a labelled negative the pipeline was otherwise discarding. On 2021+ (not used in its fit) it scores AUC 0.813 against 0.735 for a run-length-only null — and beating that null is the entire test, because "runs that have lasted longer tend to continue" is just arithmetic.

The industry grain — and its limit. The industry probability often crosses its threshold before the sector probability does, but only because it sits above that threshold on about half of all days: it rarely turns off. The aggregate industry probability is descriptive, not predictive — it is elevated roughly half the time and carries essentially no forward lift on its own, so we publish it as texture, never as a starting-gun. The genuinely useful form of "which rotation is starting" is per-industry: naming the specific industries heating up right now. That is the direction we are validating next, and every per-industry signal is held to the same 13F backtest before it earns a place on the page.

Regime metrics — and how we backtested them

The reading above is one model. On the dashboard we also publish a panel of recognized rotation and regime measures, each computed from the same survivorship-free sector data and shown as its percentile versus its own history — so you can watch the gauges you trust rather than a single number:

• Rotation Intensity — volume-spike AND turbulence — the 13F-backtested winner (ρ +0.51 vs 13F)
• Turbulence — covariance-adjusted unusual moves (Chow/Kritzman) (ρ +0.43 vs 13F)
• RRG spread — dispersion of sector relative-strength (de Kempenaer) (ρ +0.41 vs 13F)
• Volume intensity — abnormal volume — the one input independent of price (ρ +0.35 vs 13F)
• Dispersion — how far apart sector returns spread (ρ +0.29 vs 13F)
• Leadership turnover — churn in the top-3 sectors (ρ -0.34 vs 13F)
• Absorption ratio — PCA coupling — low = rotation-friendly (Kritzman) (ρ +0.08 vs 13F)
• Avg correlation — realized pairwise — low = decoupled (ρ +0.05 vs 13F)

Why we backtested, and how. A rotation gauge earns its place only if it matches rotation that actually happened, so we built an independent yardstick from ground truth: the sector bets institutions actually made. From ~580 funds' quarterly 13F filings, 2013-2026 we reconstructed the real quarter-by-quarter rotation the crowd executed — the change in each sector's share of managers' consolidated holdings, net of the drift prices alone would cause — and scored each measure by how closely it tracked that real rotation (Spearman ρ) across 51 quarters. The ρ vs 13F shown on each gauge is that score.

To keep the search honest we held out-of-sample across 5 train/test splits (test rho 0.69-0.75), and we ran a shuffle-null — letting the threshold optimizer loose on scrambled targets to measure how much a good-looking score is pure luck. The winner, Rotation Intensity (days carrying both a volume spike and elevated turbulence), cleared a shuffle-null (p=0.014); a 3-signal gate overfit (train up, test down), tracking the real 13F rotation at ρ +0.51 — against the model's +0.33.

The model and Rotation Intensity — how they relate. They are not two estimates of the same thing. The model is a calibrated probability of a rotation regime (a state); Rotation Intensity is a percentile of realized rotation magnitude versus history, from volume and turbulence — inputs the model never sees. Because they are independent, agreement is genuine corroboration and divergence is diagnostic, not a bug: a calm model with elevated intensity is often realized churn running ahead of the regime signature, while an elevated model with quiet intensity is a dispersion pattern not yet backed by heavy volume. We deliberately do not collapse them into one number — a naive combination (gating the model on volume) tracked the actual 13F rotation worse than either alone (ρ +0.28 vs +0.51), and the threshold search, given the option, declined to gate on the model. Publishing both, each with its score, is the honest resolution.

What we claim — and don't

Research and educational content only. Not investment advice, not an offer or solicitation, and not a recommendation to buy or sell any security.