research(intraday): de-bias del lead ERM (filone B) — falso positivo + gate selection-on-holdout
L'analisi di robustezza affonda lo "earns_slot=True" di ERM: era prodotto da selezione-sull'hold-out + coda 2026 + multiple-testing non corretto. A) deflated-Sharpe FAIL: 0.00 (tutti 122 trial) / 0.16 (no-TOD) / 0.24 (solo-ERM) << 0.95 B) selezione in-sample-only -> ALTRA cella (long-flat, corr->TP01 0.53) = NEUTRAL, no slot C) ensemble del plateau (no cherry-pick) -> ADDS ma robust_oos=False -> no slot D) uplift FULL solo +0.10, negativo 2021/2022; uplift HOLD +0.30 concentrato nel 2026 => ERM SCARTATO come sleeve. Conferma ennesima del soffitto BTC/ETH-direzionale ~1.3. Lezione CODIFICATA in altlib (LESSON 4, test in tests/test_harness_realism.py): - deflated_sharpe() Bailey & Lopez de Prado, PASS >= 0.95 - select_cell_insample() scelta cella col solo Sharpe pre-HOLDOUT (no peeking) - study_family_honest() gate combinato: earns_slot[cella in-sample] AND DSR>=0.95 Regola: una strategia direzionale grid-searched si giudica con study_family_honest, non chiamando study_marginal sulla cella a max hold-out. Verificato end-to-end su ERM (earns_slot_honest=False). Chiude il punto cieco gemello di CC01. Diario aggiornato (verdetto downgrade), CLAUDE.md aggiornato. Test 119/119 verdi. Nessun impatto live (branch separato). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -81,3 +81,58 @@ def test_causality_flags_lookahead():
|
||||
return f
|
||||
r = al.causality_ok(leaky, tf="1h")
|
||||
assert r["ok"] is False
|
||||
|
||||
|
||||
# --- LESSON 4: selection-on-holdout gate (codified 2026-06-29, filone B) ----------------
|
||||
def _mom_factory():
|
||||
"""Tiny continuous momentum factory parametrized by SMA lookback (for grid tests)."""
|
||||
def factory(tf, win):
|
||||
def fn(df):
|
||||
c = df["close"].values.astype(float)
|
||||
return al.vol_target(np.tanh(3 * (c / al.sma(c, win) - 1)), df, 0.20, 30, 2.0)
|
||||
return fn
|
||||
return factory
|
||||
|
||||
|
||||
def test_deflated_sharpe_penalizes_multiple_testing():
|
||||
"""The SAME Sharpe deflates toward 0 when it was the best of MANY wide-dispersion trials,
|
||||
but survives when it stood alone among a few tight ones (Bailey & Lopez de Prado)."""
|
||||
rng = np.random.default_rng(0)
|
||||
T = 1500
|
||||
idx = pd.date_range("2020-01-01", periods=T, freq="D", tz="UTC")
|
||||
sr_target = 1.0
|
||||
ret = pd.Series(0.01 * (sr_target / np.sqrt(365.25) + rng.standard_normal(T)), index=idx)
|
||||
sr_ann = al._sh(ret)
|
||||
many = list(np.linspace(-3.0, 2.5, 120)) # 120 wide-spread trials
|
||||
few = [0.1, 0.0, -0.1, 0.05] # 4 tight trials
|
||||
dsr_many, sr0_many = al.deflated_sharpe(sr_ann, many, ret)
|
||||
dsr_few, sr0_few = al.deflated_sharpe(sr_ann, few, ret)
|
||||
assert sr0_many > sr0_few # more/wider search -> higher null max
|
||||
assert dsr_many < dsr_few # multiple-testing is penalized
|
||||
assert dsr_many < 0.5 and dsr_few > 0.8
|
||||
assert 0.0 <= dsr_many <= 1.0 and 0.0 <= dsr_few <= 1.0
|
||||
|
||||
|
||||
def test_select_cell_insample_ranks_by_insample_only():
|
||||
"""The cell chosen must be the IN-SAMPLE (<HOLDOUT) Sharpe leader — never the hold-out's —
|
||||
and every searched cell's FULL Sharpe is returned for deflation."""
|
||||
factory = _mom_factory()
|
||||
grid = [dict(win=50), dict(win=150)]
|
||||
sel = al.select_cell_insample(factory, grid, ("12h",))
|
||||
assert sel["chosen"] is not None
|
||||
assert len(sel["all_full_sharpe"]) == len(grid) # one trial per (tf,cell)
|
||||
best_is = max(r["insample_sharpe"] for r in sel["rows"])
|
||||
assert sel["chosen"]["insample_sharpe"] == best_is # picked by in-sample, not hold-out
|
||||
|
||||
|
||||
def test_study_family_honest_contract():
|
||||
"""earns_slot_honest is exactly (in-sample-picked cell earns_slot) AND (deflated-Sharpe PASS).
|
||||
Both sub-gates must be enforced; the combined flag is their conjunction."""
|
||||
factory = _mom_factory()
|
||||
grid = [dict(win=50), dict(win=150)]
|
||||
rep = al.study_family_honest("MOMTEST", factory, grid, ("12h",))
|
||||
for k in ("chosen", "earns_slot_marginal", "deflated_sharpe", "dsr_pass", "earns_slot_honest"):
|
||||
assert k in rep
|
||||
assert rep["n_cells"] == len(grid)
|
||||
assert isinstance(rep["earns_slot_honest"], bool)
|
||||
assert rep["earns_slot_honest"] == bool(rep["earns_slot_marginal"] and rep["dsr_pass"])
|
||||
|
||||
Reference in New Issue
Block a user