A stock screener displays two candidates with a score of 80. One comes from a technical-signal engine; the other from a rules-based screener that combines trend and volume conditions. They share a column, so ranking them together looks natural.

There is a hidden assumption: do both scores measure the same thing?

While organizing the design notes for a multi-source stock screening system, I found an intentional difference in how sources were scored. A composite technical signal and a screener preset do not always produce the same evidence. Penalizing one source for missing a field defined only for another source would be a category mistake. Source-aware scoring addresses that. But the resulting 0–100 numbers still do not automatically have a shared statistical meaning.

A raw score of 80 is not an 80% chance of a positive return. A sound comparison first fixes the outcome definition, then evaluates settled observations under comparable market conditions and time periods. If there is insufficient evidence, source-specific ranking is more defensible than a deceptively precise global leaderboard.

Why different signal sources need different scoring profiles

A June 2026 project-design note described a signal-quality score that combined resonance_score and technical_rating. At that point, the note used a 70/30 weighting illustration. A screener route had a different set of inputs: technical_rating, moving-average rating, RSI, MFI, distance from a 52-week high, and the number of matched filter conditions. It did not necessarily have a composite-signal-specific resonance_score.

These notes describe a historical design, not an audit of the currently deployed implementation. Their significance is architectural: absence of a source-exclusive feature should not be mistaken for poor signal quality.

Source-aware scoring solves feature-availability fairness. It does not prove comparability after scores are combined on one axis.

Suppose A’s 80 means strong agreement with its resonance rules, while B’s 80 means strong agreement with its screening rules. Shared display units do not imply shared probabilities. The scikit-learn probability calibration guide explains how to check predicted probabilities against observed event frequencies, and why calibration needs data independent of model fitting. Raw heuristic scores must not be treated as calibrated probabilities.

Even replacing raw scores with each source’s historical hit rate can fail. The problem starts one layer earlier: which observations entered the denominator?

Simpson’s paradox: both scores are 80, yet the aggregate comparison flips

To isolate that question, consider 80 deliberately constructed teaching observations (a toy example, not a live trading backtest). Every observation has raw_score = 80. There are two sources, two imagined market regimes (trend and range), and a predefined five-day binary outcome label.

The pooled column makes A look slightly better. Within each regime B performs better: 80% versus 70% in a trend, 50% versus 30% in a range. A has more observations in the easier trend regime; B has more in the harder range regime. Their mixture, not a demonstrated strategy advantage, drives the aggregate ordering.

Standardize both sources to the same hypothetical 50/50 trend/range mix:

The ordering reverses. This is the classic aggregation problem known as Simpson’s paradox. A 50/50 mix was chosen for the illustration, not estimated from any particular market. In a real study the target population and its weights must be specified before looking at which source wins.

Arithmetic is not the most useful part to automate. A more valuable question is how misleading a high rate becomes in a tiny bucket. B’s trend bucket has eight positives in ten cases: an apparent 80%. Under an independent Bernoulli assumption, the 95% Wilson lower confidence bound is only about 49.0%.

import math

def wilson_lower_bound(wins: int, n: int, z: float = 1.96) -> float:
    if n <= 0 or not 0 <= wins <= n:
        raise ValueError("invalid wins or sample size")
    p = wins / n
    denom = 1 + z*z/n
    center = p + z*z/(2*n)
    margin = z * math.sqrt(p*(1-p)/n + z*z/(4*n*n))
    return (center - margin) / denom

print(f"8/10 = 80%; Wilson 下界 = {wilson_lower_bound(8, 10):.1%}")
# 8/10 = 80%; Wilson 下界 = 49.0%

Notice the deliberate exception for n = 0: a bucket with no observations has no evidence, not a measured 0% success rate. The Wilson interval itself is not a guaranteed conservative trading win rate. Repeated appearances of the same stock and overlapping five-day outcome windows break simple independence assumptions; they require grouping or dependence-aware uncertainty estimates.

#!/usr/bin/env python3
"""Deterministic synthetic check of group-conditional scoring; NOT a market backtest.

All labels below are intentionally constructed. There are no real stock prices,
no actual trading outcomes, and no production scoring-engine output.
"""
import json
import math
from collections import defaultdict

# Fixed illustrative counts: (source, hypothetical regime, observations, positive outcomes)
SPEC = [
    ("technical_signal", "trend", 30, 21),
    ("technical_signal", "range", 10, 3),
    ("screener_preset", "trend", 10, 8),
    ("screener_preset", "range", 30, 15),
]


def wilson_interval(positive, total, z=1.959963984540054):
    if total < 1:
        raise ValueError("empty group")
    p = positive / total
    denom = 1 + z*z/total
    center = (p + z*z/(2*total)) / denom
    half = z * math.sqrt(p*(1-p)/total + z*z/(4*total*total)) / denom
    return [round(center-half, 4), round(center+half, 4)]


def make_group(total, positive):
    return {"n": total, "positive": positive, "rate": round(positive / total, 4),
            "wilson_95": wilson_interval(positive, total)}


def main():
    rows = []
    for source, regime, total, positive in SPEC:
        rows.extend({"source": source, "regime": regime, "raw_score": 80,
                     "positive_event_5d": int(i < positive)} for i in range(total))
    summaries = defaultdict(dict)
    pooled = defaultdict(lambda: [0, 0])
    for source, regime, n, yes in SPEC:
        summaries[source][regime] = make_group(n, yes)
        pooled[source][0] += n
        pooled[source][1] += yes
    result = {"scope": "CONSTRUCTED_SYNTHETIC_EXAMPLE_NOT_MARKET_DATA",
              "event_definition": "binary illustrative positive_event_5d, not executed trade PnL",
              "raw_score": 80, "total_rows": len(rows), "sources": {}}
    for source in sorted(summaries):
        trend = summaries[source]["trend"]["rate"]
        range_ = summaries[source]["range"]["rate"]
        n, yes = pooled[source]
        result["sources"][source] = {
            "by_regime": summaries[source], "pooled": make_group(n, yes),
            "equal_regime_mix_rate": round((trend + range_) / 2, 4)
        }
    a = result["sources"]["technical_signal"]
    b = result["sources"]["screener_preset"]
    assert len(rows) == 80
    assert a["pooled"]["rate"] == 0.6 and b["pooled"]["rate"] == 0.575
    assert a["equal_regime_mix_rate"] == 0.5 and b["equal_regime_mix_rate"] == 0.65
    assert all(b["by_regime"][r]["rate"] > a["by_regime"][r]["rate"] for r in ("trend", "range"))
    result["test_assertions"] = {
        "fixed_80_records": "PASS", "pooled_order_a_above_b": "PASS",
        "within_each_regime_b_above_a": "PASS", "fixed_equal_mix_order_b_above_a": "PASS"
    }
    print(json.dumps(result, ensure_ascii=False, indent=2, sort_keys=True))


if __name__ == "__main__":
    main()

The complete standard-library script creates the 80 illustrative rows, reports regime and pooled proportions and Wilson intervals, then verifies four assertions. Its purpose is to catch the aggregation mechanism, not to establish which trading engine is more profitable.

Real data makes the issue harder. The same stock can trigger two sources on the same date; one source can favor trends while the other favors pullbacks. Signals dated t and t+1 have overlapping five-day forward returns. Adding such records inflates row counts much faster than it increases independent evidence.

Define the event before you calibrate any score

Before choosing logistic or isotonic calibration, write down what “success” means. A positive close-to-close return after five trading sessions, outperforming an index, and an executable net-profit trade are not interchangeable outcomes. Neither are one-day and ten-day labels.

For a narrow research exercise, one possible target is: at a fixed market close and decision timestamp, classify whether the five-session return exceeds zero after a specified transaction-cost model. Only score the event once its label has matured. If actual execution prices are unavailable, call it a price-return proxy, not realized trading P&L.

Keep the decision-time snapshot immutable:

decision_id / symbol / trade_date / as_of_time
source_key / raw_score / model_version
market / session / regime
horizon_days / label_status / realized_outcome

Each field closes a different loophole. source_key identifies the scoring profile; model_version stops weights from silently being revised in historical comparisons; as_of_time ensures the inputs were known at the decision. Recomputing last month’s “historical” scores with this month’s data and model code is not point-in-time validation.

One security appearing under two source_keys at one timestamp can be kept twice to study source attribution. It must not be counted as two independent market outcomes when estimating uncertainty or portfolio performance. The analysis must declare its decision-level deduplication and clustering policy.

Pending labels need equal care. An outcome that has not reached day five is neither a failure nor a zero return. Comparing only completed observations between sources of different ages introduces maturity selection bias. Different universes, delisted instruments, reporting lags, adjusted prices, and liquidity constraints add further differences in what each source actually samples.

These may look like bookkeeping problems. In practice, they decide whether any calibration curve has a meaning worth testing.

What calibration can establish—and what it cannot

Start by bucketing raw scores by source and time period. Report the observation count, settlement coverage, event rate, return distribution, and uncertainty for each relevant market/regime/session grouping. Then decide whether there is enough information to fit any mapping to event probabilities.

Sparse cells are a reason to keep source-specific ranks or consider pooled hierarchical estimates with shrinkage, not to train a separate flexible calibrator for every source × regime × session combination. An apparent 100% rate in a tiny bucket is usually a warning about sample size.

With sufficient settled data, compare simple sigmoid calibration with more flexible methods. Evaluate on independent later observations, not the same events used to fit the mapping. A reliability diagram compares the forecast probability in a bucket with its observed positive fraction. Brier Score also reflects discrimination and outcome uncertainty; a lower score does not prove that a trading system is more profitable.

Time-series leakage requires another check. A signal at t might not get its five-day label until t+5. If a training sample just before a test boundary uses price movements that occur during the test window, the model can have advance knowledge even when its signal date precedes the test dates. TimeSeriesSplit supports chronological splits and a gap, but “gap=5” is not a universally safe embargo; event completion times, calendars, same-stock overlap, and holding periods matter.

Do not combine two separate goals into an ambiguous “accuracy”:

A model can rank well without being calibrated. A probability predictor may match a group’s base rate while contributing almost no security-level discrimination. Google’s prediction bias discussion offers a useful average-prediction versus average-label sanity check, but matching aggregates is not proof of calibration within every score bin or source.

And probability is not payoff. Even a well-calibrated probability of a positive five-day return says nothing by itself about how large gains are, how severe losses become, execution slippage, drawdowns, or expected returns. A high hit rate can coexist with a poor reward-to-risk ratio.

Keep source-specific scores until cross-source evidence earns its place

I would not rush to re-rank every source into one universal leaderboard. Different profiles are useful: composite signals may rely on resonance, while screeners rely on their own conditions. The missing piece is the evidence needed for cross-source interpretation.

My next read-only research pass would select ordinary settled batches, exclude test and manual verification batches, freeze each source_key and model_version, match the market/session/horizon, and compare score buckets by regime. I would inspect median returns, adverse tails, duplicate instruments, coverage, and maturity before evaluating a calibration mapping on later dates.

When the available evidence is thin, show raw_score together with source identity and sample coverage rather than inventing a precise unified probability.

This question complements the earlier Bucket Spread and Rank IC factor validation and the study of regime-specific weighting and shadow activation. The first asks whether a factor contains ranking information; the other asks whether changed weights are ready to go live. This one sits between them: can two scoring systems’ idea of a “strong” candidate be compared at all?

Actual source rankings still require matched, settled, out-of-time outcome evidence. Until those results exist, retaining the original score and its source is more useful than printing a suspiciously exact “universal win probability” beside every stock.

When you see two stocks both scoring 80, ask what was scored, which event was labeled, and who entered the sample—before assuming the numbers belong on the same axis.