How the rotation reading is built

The question

Formally this is a state-estimation problem. The latent state is rotation phase; the observables are noisy sector-level returns and breadth. The deliverable is a calibrated posterior probability over rotation phase at daily frequency, using data through today only — with two kinds of noise removed first: market-wide movement, and the distortion from a single dominant constituent (a cap-weighted sector index is hostage to its largest holding).

Data — owned, not borrowed

The project owns its price feed on one consistent, single-source basis rather than reading a shared, multi-vendor table. That discipline is deliberate — mixing vendors with no cross-source resolution lets one bad print corrupt returns invisibly. Each return is computed within a single source; where two sources meet, they are joined by return, not by splicing incompatible levels.

• Prices — 6.97M daily bars from a delisting-complete vendor, 1990-2026. The companies that left the index are still present, so the dataset is free of survivorship bias — the single most important property for this question, because names that leave the index are systematically the worst performers and cluster in exactly the sectors rotating out.
• Membership & weights — S&P 500 constituents and float weights as they actually stood each quarter, joined by ISIN, never ticker (tickers silently recycle — the same symbol can point at two different companies over time).
• Cross-checks — prices reconcile to a second independent feed at 99.85% agreement; cap-weighted sector returns reconcile to the SPDR sector ETFs at 0.978 correlation. Disagreements are flagged, never silently overwritten.

Denoising

Before modelling, each sector return is taken net of its own trailing beta to the market, leaving a market-denoised residual. Alongside it we compute breadth (how many members participate) and concentration (how much of the move is one name). A genuine rotation shows broad participation; a fake one driven by a single mega-cap does not — and the breadth/concentration measures are the direct discriminators.

The mechanical ruler

To detect a rotation you must first define one. The definition is pre-registered and frozen before any model is fit: cross-sectional dispersion above a trailing percentile, for a minimum run of days, with sufficient breadth in the leading sectors and low rank-correlation to the prior ordering. This mechanical rule is an external reference the model never sees — so "state 2 is a rotation" becomes checkable, not asserted.

The model

The reading comes from a discrete hidden Markov model (three states) over the sector observables. Two disciplines matter most:

• No look-ahead. The probability at each date uses data through that date only — a filtered estimate. Smoothed (full-sample) states look dramatically better and are useless in real time; they are never used to score anything.
• Calibration. The raw model score is mapped to an honest probability by isotonic regression fit on out-of-sample data, so "0.30" means what it says.

The model is unsupervised — it never sees the mechanical ruler. That one of its states independently lines up with the ruler is the evidence that the state is real rather than an artifact of fitting.

Validation

Rotations are rare, so a single train/test split cannot confirm anything. The evidence is built in two stages:

• Blocked, purged cross-validation across 1990-2026 (the primary evidence) — contiguous blocks with a gap at each boundary so nothing leaks through the trailing windows. Median out-of-sample AUC 0.909.
• A single sealed-holdout test on 2021+, run exactly once with everything frozen on the earlier data: AUC 0.8722, calibration error 0.0041.

Every result is reported against honest benchmarks — the unconditional base rate (14.1%), the mechanical rule alone, and a simple dispersion threshold — so the latent-state machinery has to earn its complexity. The model learned from 104 rotation episodes, spanning every regime a reader would name.

What we claim — and don't

Research and educational content only. Not investment advice, not an offer or solicitation, and not a recommendation to buy or sell any security.