When you build a stock-ranking or scoring model, adding another factor is the easy part.

Momentum, RSI, relative strength, distance from a 52-week high, volatility, volume—if something looks vaguely useful in a backtest, it is tempting to add another column and give it a weight. The harder question comes later: is this factor actually ranking future returns, or did it just look good in one window?

While cleaning up my own quant-research workflow, I noticed how easy it is to get pulled toward win rate or a single forward-return horizon. A factor can look good five days later while behaving very differently after one or three days. And if you evaluate the same signals every day, the sample count grows quickly even though forward returns overlap heavily, so the apparent amount of independent evidence can be much smaller than the row count suggests.

Instead of reusing old IC or spread numbers from my internal notes, I reran a small public reproduction. The data comes from five ETFs in a public repository: sector_prices.csv. I pinned the file at blob SHA 0a442a9260c2ad64e6cd8c01af49328eb7619367, used SPY, XLK, XLF, XLE, and XLV, built a few simple 20-day price factors, and measured 1D / 3D / 5D forward returns.

The result was not a neat “best factor.” That was the useful part.

For 20-day momentum, the high-minus-low median spread was positive at 1D and 5D in the sparse-anchor sample, but slightly negative at 3D. The low-volatility factor also changed direction across horizons. More interestingly, a “20-day relative strength versus SPY” factor ended up producing exactly the same cross-sectional ranking as raw 20-day momentum.

Those messy edges are much closer to real research than an all-green factor report.

Freeze the factor value at the time of the signal

The most important rule in factor validation is not the formula. It is the time boundary.

Suppose the system scores a group of stocks on a given day and records momentum_20d. What you want to save is the value that existed when the decision was made, then wait for the 1D, 3D, and 5D outcomes to settle and evaluate whether that value had ranking power.

In other words:

decision / snapshot -> factor value at t -> realized return at t + N

What you do not want is to wait three days, then recompute the “historical” factor using today’s data and today’s code.

Those two approaches look similar, but they are not equivalent. A data vendor can revise history. Corporate-action adjustments can change. The asset universe can change. The factor definition can change. Cross-sectional normalization can use a different set of securities. Once you recompute later, you risk validating “today’s understanding of the past” instead of what the system actually saw at the time.

Alphalens follows the same basic shape: align factor values with later forward returns, analyze returns by quantile, and calculate an Information Coefficient. Its documentation also calls out a very concrete look-ahead risk: filtering forward returns with a z-score calculated from those same future returns leaks future information into the cleaning step.

If I eventually let my production research system adjust weights automatically, I want this boundary to be stricter than it is in an ordinary notebook: once a factor snapshot enters outcome tracking, later model versions should not be able to rewrite it.

The public reproduction here does not use my production database. I construct signal-time factors and forward returns from one pinned historical-price file. That is enough to verify the calculation flow, but not enough to prove that any production system is free of point-in-time-data problems. Those are separate claims.

Bucket Spread and Rank IC answer different questions

If a factor is mainly used for ranking, I do not like starting with “what is the win rate?”

Win rate compresses too much information into a Boolean. A +0.01% return and a +8% return are both “wins.” A stock ranked first and one ranked fifth may both be counted the same way. Win rate can describe a strategy outcome, but it does not directly answer whether higher factor values correspond to better future outcomes.

I used two basic measurements.

The first is Bucket Spread. On each signal date, rank the cross-section by the factor, split it into Low / Mid / High groups, and compare the later returns of the High and Low groups:

Bucket Spread = median(return_high) - median(return_low)

If a “higher is better” factor contains ranking information, you would hope the High group outperforms the Low group. But a positive spread only says that the two ends separated in this sample. It does not automatically imply statistical significance, it does not mean the edge survives trading costs, and it does not guarantee that the middle bucket is monotonic.

The second is Rank IC. I use Spearman rank correlation between factor rank and future-return rank:

Rank IC = Spearman(factor_rank, forward_return_rank)

SciPy’s spearmanr is a nonparametric measure of monotonic association, ranging from -1 to 1. For many factors, that is a natural fit: I care whether the ordering contains information, not whether factor values and future returns have a linear relationship.

The core calculation in this experiment is deliberately small:

mom20 = px / px.shift(20) - 1
fwd = px.shift(-horizon) / px - 1

x = mom20.loc[dt].dropna()
y = fwd.loc[dt].dropna()
common = x.index.intersection(y.index)

bucket = pd.qcut(
    x.loc[common].rank(method="first"),
    3,
    labels=["low", "mid", "high"],
)

rho, _ = spearmanr(
    x.loc[common].values,
    y.loc[common].values,
)

spread = (
    y.loc[common][bucket == "high"].median()
    - y.loc[common][bucket == "low"].median()
)

The actual script is pinned in the Content Lab evidence, and the executed command exited with code 0. I did not build a new quant framework here. I reduced the test to the smallest version that still exposes the core validation logic.

Change the horizon, and the story starts to move

To reduce forward-return overlap, I use every fifth trading day as the main display sample. That leaves 14 anchor dates. With five ETFs per date, each factor × horizon combination contains 70 cross-sectional observations.

The spread below is High-bucket median minus Low-bucket median, in percentage points. Rank IC is the mean cross-sectional Spearman IC across the anchor dates.

Factor	Horizon	N	Median Spread	Mean Rank IC
20D Momentum	1D	70	+0.583 pp	+0.179
20D Momentum	3D	70	-0.071 pp	+0.014
20D Momentum	5D	70	+0.667 pp	+0.214
Distance to 20D High	1D	70	+0.249 pp	+0.089
Distance to 20D High	3D	70	+0.259 pp	+0.151
Distance to 20D High	5D	70	+0.529 pp	+0.252
Low Volatility 20D	1D	70	-0.029 pp	+0.029
Low Volatility 20D	3D	70	+0.296 pp	+0.014
Low Volatility 20D	5D	70	-0.224 pp	-0.207
Relative Strength 20D vs SPY	1D	70	+0.583 pp	+0.179
Relative Strength 20D vs SPY	3D	70	-0.071 pp	+0.014
Relative Strength 20D vs SPY	5D	70	+0.667 pp	+0.214

The thing I least want to do with this table is circle the largest number and declare the factor “valid.”

Take 20D Momentum. If you only look at 5D, a +0.667 pp spread and +0.214 mean Rank IC make it very tempting to increase its weight. But the 3D spread is -0.071 pp and the Rank IC is close to zero. There is no clean “the longer the horizon, the better the factor” pattern.

Low Volatility is even messier. Its 3D median spread is positive, while the 5D spread flips to -0.224 pp and mean Rank IC falls to -0.207. If I only showed the 3D column, it would look like a promising positive factor. At 5D, the story is different.

That is the practical value of multi-horizon outcome tracking. It is not primarily about mixing 1D, 3D, and 5D into a prettier weighted score. It is about exposing how sensitive the conclusion is to the horizon.

I also kept a daily-overlapping sample. That version has 66 signal dates and 330 factor/outcome rows, so the headline sample size looks much larger.

The problem is that 3D and 5D forward returns overlap heavily.

The result actually changes: the 3D median spread for 20D Momentum is about +0.578 pp in the daily-overlapping sample, but -0.071 pp in the every-fifth-trading-day anchor sample.

I do not treat that difference as proof that one sampling scheme is “the truth.” Fourteen anchors are still a small sample, and five-day spacing does not remove every dependency. It supports a narrower lesson: if a factor’s conclusion can flip when you change a reasonable sampling scheme, a research report should not hide that behind one large total-row count.

Before adding more factors, check whether they carry the same ranking information

One result initially made me think I had a bug.

I tested both:

momentum_20d
relative_strength_20d_vs_spy

The second factor is:

asset_20d_return - SPY_20d_return

After running the analysis, every bucket, spread, and Rank IC value was identical.

The code was fine.

On a given date, SPY’s 20-day return is the same constant for every asset:

RS_i = Momentum_i - C_t

Subtracting the same C_t from every asset in the cross-section does not change the ordering. If Bucket and Rank IC depend only on cross-sectional rank, then these two definitions contain the same ranking information in this setup.

That was more useful to me than seeing “IC = 0.21” for another factor.

A multi-factor model can accumulate fake complexity very easily: different names, different-looking formulas, but essentially the same directional information. Giving two highly redundant factors independent weights does not create two pieces of evidence. It lets one piece of evidence vote twice.

This does not mean “relative strength equals momentum” in general. Relative strength can be defined versus an industry group, a dynamic benchmark, residual returns, rolling-beta-adjusted returns, or a different lookback window. Any of those can change the ranking. The equivalence here comes from one specific construction: subtracting the same SPY 20D return from every asset on the same date.

The habit I want to keep is simple: before tuning weights, compare factor ranks, correlations, and bucket assignments. Check whether the new factor adds information or only adds a new label.

I would not let one positive IC change live weights

If a research system will eventually tune a model automatically, I prefer a gap between “factor discovered” and “factor changes production.”

A candidate factor can start in observe mode: record the point-in-time value, but do not use it in decisions. Once outcomes settle, inspect Bucket Spread, Rank IC, coverage, and sample distribution across horizons.

If the signal still looks interesting, move it into shadow mode. Shadow can produce a candidate weight set or hypothetical decisions, but it still does not modify the active model.

At minimum, I want to see:

• enough sample size and non-null coverage;
• some sensible Low / Mid / High ordering, not only a lucky difference between the extremes;
• 1D / 3D / 5D behavior that can be explained rather than mutually contradictory;
• whether the direction survives a different period or market regime;
• whether the factor is highly redundant with existing factors;
• whether changing the sampling or deduplication rule changes the conclusion;
• how many factors, windows, and parameter choices were tried during the research process.

That last point matters a lot. Factor research often becomes a large set of hypothesis tests. If you try 50 indicators, six lookback windows, and four forward horizons, then pick the nicest result, the probability of finding something attractive increases even when the underlying signal is weak. Harvey, Liu, and Zhu discuss this problem directly in … and the Cross-Section of Expected Returns.

So I do not treat Bucket Spread and Rank IC as a factor admission license. They are closer to basic diagnostics. They can show whether the ordering contains information, how stable the direction is, and whether horizon flips or factor redundancy exist. But there is still a large gap between that and “this factor is tradable with real money”: stability, transaction costs, out-of-sample validation, regime dependence, portfolio exposure, and risk controls all remain.

This public experiment also uses only five ETFs and 14 sparse anchor dates. It does not prove any production alpha. What it does prove is that the validation chain can be reproduced in a small form—and that once you expose horizon, sampling, and rank redundancy, many clean-looking factor conclusions become much less certain.

That uncertainty is worth keeping.

If a factor deserves to enter a model, I would rather see it survive a long period of outcome tracking and several uncomfortable checks than trust it because one research report produced a pretty number.