research(intraday): de-bias del lead ERM (filone B) — falso positivo + gate selection-on-holdout

L'analisi di robustezza affonda lo "earns_slot=True" di ERM: era prodotto da
selezione-sull'hold-out + coda 2026 + multiple-testing non corretto.
  A) deflated-Sharpe FAIL: 0.00 (tutti 122 trial) / 0.16 (no-TOD) / 0.24 (solo-ERM) << 0.95
  B) selezione in-sample-only -> ALTRA cella (long-flat, corr->TP01 0.53) = NEUTRAL, no slot
  C) ensemble del plateau (no cherry-pick) -> ADDS ma robust_oos=False -> no slot
  D) uplift FULL solo +0.10, negativo 2021/2022; uplift HOLD +0.30 concentrato nel 2026
=> ERM SCARTATO come sleeve. Conferma ennesima del soffitto BTC/ETH-direzionale ~1.3.

Lezione CODIFICATA in altlib (LESSON 4, test in tests/test_harness_realism.py):
  - deflated_sharpe()       Bailey & Lopez de Prado, PASS >= 0.95
  - select_cell_insample()  scelta cella col solo Sharpe pre-HOLDOUT (no peeking)
  - study_family_honest()   gate combinato: earns_slot[cella in-sample] AND DSR>=0.95
Regola: una strategia direzionale grid-searched si giudica con study_family_honest,
non chiamando study_marginal sulla cella a max hold-out. Verificato end-to-end su ERM
(earns_slot_honest=False). Chiude il punto cieco gemello di CC01.

Diario aggiornato (verdetto downgrade), CLAUDE.md aggiornato. Test 119/119 verdi.
Nessun impatto live (branch separato).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Adriano Dal Pastro
2026-06-29 20:04:36 +00:00
parent aad69f9790
commit b4ec92734c
5 changed files with 471 additions and 9 deletions
+55
View File
@@ -81,3 +81,58 @@ def test_causality_flags_lookahead():
return f
r = al.causality_ok(leaky, tf="1h")
assert r["ok"] is False
# --- LESSON 4: selection-on-holdout gate (codified 2026-06-29, filone B) ----------------
def _mom_factory():
"""Tiny continuous momentum factory parametrized by SMA lookback (for grid tests)."""
def factory(tf, win):
def fn(df):
c = df["close"].values.astype(float)
return al.vol_target(np.tanh(3 * (c / al.sma(c, win) - 1)), df, 0.20, 30, 2.0)
return fn
return factory
def test_deflated_sharpe_penalizes_multiple_testing():
"""The SAME Sharpe deflates toward 0 when it was the best of MANY wide-dispersion trials,
but survives when it stood alone among a few tight ones (Bailey & Lopez de Prado)."""
rng = np.random.default_rng(0)
T = 1500
idx = pd.date_range("2020-01-01", periods=T, freq="D", tz="UTC")
sr_target = 1.0
ret = pd.Series(0.01 * (sr_target / np.sqrt(365.25) + rng.standard_normal(T)), index=idx)
sr_ann = al._sh(ret)
many = list(np.linspace(-3.0, 2.5, 120)) # 120 wide-spread trials
few = [0.1, 0.0, -0.1, 0.05] # 4 tight trials
dsr_many, sr0_many = al.deflated_sharpe(sr_ann, many, ret)
dsr_few, sr0_few = al.deflated_sharpe(sr_ann, few, ret)
assert sr0_many > sr0_few # more/wider search -> higher null max
assert dsr_many < dsr_few # multiple-testing is penalized
assert dsr_many < 0.5 and dsr_few > 0.8
assert 0.0 <= dsr_many <= 1.0 and 0.0 <= dsr_few <= 1.0
def test_select_cell_insample_ranks_by_insample_only():
"""The cell chosen must be the IN-SAMPLE (<HOLDOUT) Sharpe leader — never the hold-out's —
and every searched cell's FULL Sharpe is returned for deflation."""
factory = _mom_factory()
grid = [dict(win=50), dict(win=150)]
sel = al.select_cell_insample(factory, grid, ("12h",))
assert sel["chosen"] is not None
assert len(sel["all_full_sharpe"]) == len(grid) # one trial per (tf,cell)
best_is = max(r["insample_sharpe"] for r in sel["rows"])
assert sel["chosen"]["insample_sharpe"] == best_is # picked by in-sample, not hold-out
def test_study_family_honest_contract():
"""earns_slot_honest is exactly (in-sample-picked cell earns_slot) AND (deflated-Sharpe PASS).
Both sub-gates must be enforced; the combined flag is their conjunction."""
factory = _mom_factory()
grid = [dict(win=50), dict(win=150)]
rep = al.study_family_honest("MOMTEST", factory, grid, ("12h",))
for k in ("chosen", "earns_slot_marginal", "deflated_sharpe", "dsr_pass", "earns_slot_honest"):
assert k in rep
assert rep["n_cells"] == len(grid)
assert isinstance(rep["earns_slot_honest"], bool)
assert rep["earns_slot_honest"] == bool(rep["earns_slot_marginal"] and rep["dsr_pass"])