diff --git a/CLAUDE.md b/CLAUDE.md index da019bd..7c00008 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -40,6 +40,14 @@ Prima ondata di ricerca onesta su BTC/ETH certificati (5 track, harness condivis Deploy/paper a **1d**. Diari `2026-06-19-tp01-verification.md` / `-tp01-lookahead-fix-lf.md`. Paper trader: `scripts/live/paper_trend.py` (1d). Test: `tests/test_trend_portfolio.py`. Ri-verifica: `scripts/analysis/{verify_tp01,stress_tp01,tp01_lowfreq}.py`. + ⚠️ **ANCHOR TIMING-LUCK (2026-07-02, confermato da scettico):** l'hold-out ~0.31 è calcolato + sull'ancora daily 00:00 UTC, che è la **migliore delle 24 possibili** (mediana ancore 0.04, banda + [−0.13,+0.30]; P~0.86 che una qualsiasi ancora mostri uno spike così per puro caso) → l'hold-out + 2025-26 NON risolve l'edge di ritorno di TP01; ciò che regge a OGNI ancora è il **taglio del DD** + (7-10% vs ~60% B&H). FULL/plateau/deflated-Sharpe/gate INVARIATI (h=0 al 31° pctl su FULL). + Regola: i futuri numeri hold-out di strategie a ribilanciamento ancorato si citano con la banda + d'ancora. Diario `2026-07-02-timing-crt-wave.md`; script `scripts/research/r0702_tp01_offset.py` + + `r0702_skeptic_offset.py`. - **XS01 Cross-Sectional Momentum (Hyperliquid) — DIVERSIFICATORE che migliora il portafoglio** — `src/portfolio/sleeves.py:_xsec_returns`. Market-neutral su **19 alt liquidi major** Hyperliquid (1d, dal 2024): ogni 10g long i 5 più forti / short i 5 più deboli, vol-target 20%. **Scorrelato a TP01 @@ -190,6 +198,32 @@ Prima ondata di ricerca onesta su BTC/ETH certificati (5 track, harness condivis tz-aware non-ns (pandas 2.x) → scala sbagliata → `merge_asof` broadcasta = **look-ahead che `causality_ok` non vede**; usare epoca esplicita in ms (altlib verificato pulito). Diario di sintesi `2026-07-01-strategy-wave-6threads.md` + 6 diari di filone; script `scripts/research/r0701_*.py`. +- **Ondata 2026-07-02 (TIMING + CRT, 8 filoni multi-agente + scettico) — 0 nuovi sleeve, 1 finding + strutturale (anchor timing-luck di TP01, vedi ⚠️ nel bullet TP01).** Goal: "strategie con timing + differenti". (1) **Event-clock bars** (volume/vol/range da 5m, TSMOM/Donchian/EWMA in tempo-informazione): + batte il wall-clock a pari segnale/frequenza solo in 4/45 coppie; cella best IS 1.45 → HOLD −0.46, + NEUTRAL (corr 0.74 = trend travestito) → SCARTATO: il clock non è dove vive l'edge. (2) **Calendario + scadenze Deribit** (expiry weekly/monthly/quarterly ven 08:00 UTC): 0/24 celle a Bonferroni; il drift + post-expiry monthly fallisce placebo-weekday e permutation e si INVERTE sul quarterly (dove l'OI massimo + dovrebbe amplificarlo); unico pattern robusto = gio→ven negativo, ma è day-of-week (SEA morta) a Sharpe + netto ~0 → SCARTATO. (3) **Anchor timing-luck TP01 + tranching**: finding confermato (dettagli nel + bullet TP01); tranching K=2/4 = sola riduzione della varianza della STIMA (ΔSharpe n.s., ΔDD ~0.5pt), + NO deploy a $600 (il min-order lo degenera in K=1; serve feed intraday fuori path certificato) — + rivalutare a ≥5-10k; ⚠️ audit d'ancora PENDENTE su XS01 (rebalance 10g) e SKH01 (fase 230m/690m). + (4) **Clock lenti (2-7g) + bande isteresi**: fee drag di TP01 = ~0.4%/anno = tetto di ogni risparmio; + il lag costa più del risparmio (HOLD ensemble 0.34→0.11 da N=2 a 7); a $600 **il min-order $5 è GIÀ la + banda ottimale** (ordini −74% a costo ~0) → nessun cambio al book. (5) **Velocità trend + regime-condizionata** (pesi tra orizzonti 30/90/180g vs percentile vol RV/DVOL): pctl 0.71 vs null + pesi-statici-casuali = tilt-30d statico travestito (trappola EW-STR); pesi canonici 1/3 confermati → + SCARTATO. (6-8) **CRT "Candle Range Theory"** (sweep-and-reclaim 3 candele, mai coperto da MRV/MIC): + base 864 trial DSR 0.000 + anchor-flip + short "smart-money" negativo perfino in-sample; multi-TF + (4h→15m, 1h→5m, ~10k trade) expectancy negativa ovunque anche a fee zero, e **il ritest è informazione + negativa** (pattern con-ritest −40bps vs senza +52bps: aspettarlo seleziona i peggiori); contesto + (FVG/equal-highs/sessioni, 22 trial) non salva il fade, cella "Asia" = artefatto anchor-flip → + SCARTATO 3/3. Sottoprodotto: sugli stessi livelli prior-day **FOLLOW > FADE ogni anno 2019-26** + (conferma indipendente del lead prevday in forward-monitor). Lezione: il timing-luck d'ancora è + multiple-testing che il deflated-Sharpe NON conta (candidato gate futuro `anchor_luck_band`). + Diario `2026-07-02-timing-crt-wave.md`; script `scripts/research/r0702_*.py` (9 file). - **Soffitto strutturale BTC/ETH-direzionale ~1.3** superato SOLO espandendo a un meccanismo diverso: cross-sectional su universo Hyperliquid certificato (XS01) → portafoglio Sharpe ~1.55. - **Sweep "strategie alternative" (2026-06-20) — 104 ipotesi / 153 agenti / NIENTE di nuovo regge.** diff --git a/docs/diary/2026-07-02-timing-crt-wave.md b/docs/diary/2026-07-02-timing-crt-wave.md new file mode 100644 index 0000000..9021960 --- /dev/null +++ b/docs/diary/2026-07-02-timing-crt-wave.md @@ -0,0 +1,186 @@ +# 2026-07-02 — Ondata timing + CRT (8 filoni multi-agente + scettico): 0 nuovi sleeve, 1 finding strutturale (anchor timing-luck di TP01) + +**Obiettivo (goal utente):** "cerca su Deribit altre strategie profittevoli magari con timing +differenti oltre a quelle già valutate". Prima ondata interamente dedicata al TIMING: dopo che le +ondate precedenti hanno coperto segnali (104 famiglie alt-sweep), universi (HL cross-sectional), +opzioni (VRP/gamma) e pesi (EW-STR/weights_tilt_null), qui si attacca la dimensione rimasta: *quando* +si campiona, si decide e si esegue. Su richiesta utente, seconda tranche dedicata alla **CRT (Candle +Range Theory)**, il pattern sweep-and-reclaim a 3 candele di scuola ICT/Wyckoff. + +Inventario preventivo per non rifare lavoro: SEA01-09 coprivano già hour-of-day, day-of-week, +weekend, turn-of-month, sessioni; l'onda intraday 2026-06-21 il sub-daily (con il lead prevday in +forward-monitor); `tp01_lowfreq` i TF 4h/12h/1d. Angoli NON coperti individuati e testati: clock a +eventi, calendario scadenze Deribit, ancora di ribilanciamento, clock più lenti del daily, velocità +del segnale regime-condizionata — più i 3 tagli CRT (base, multi-TF, contesto). + +**Esito: 7 filoni FAIL + 1 finding strutturale confermato dallo scettico (timing-luck dell'ancora +di TP01). Nessun nuovo sleeve. Nessuna modifica al book live. Il soffitto ~1.3 direzionale +BTC/ETH resta intatto anche cambiando il clock.** + +--- + +## Filone 1 — Event-clock bars (tempo-informazione): FAIL + +`scripts/research/r0702_eventclock.py`. Volume/vol/range bars da 5m certificato (soglie causali +EWMA-90g shift(1), barre medie 4h/12h/24h), 5 strategie (TSMOM multi-orizzonte, Donchian 10/30d, +EWMA-cross), 45 celle event + 15 controlli wall-clock = 60 trial. + +- Null decisivo (stesso segnale, stessa frequenza, wall-clock): event batte wall su ENTRAMBE le + finestre solo in **4/45 coppie (9%)** — rumore. Pattern incoerenti (EWMA event vince IS 9/9 ma + perde HOLD 9/9; Donchian-30d l'esatto opposto = selection-on-holdout in agguato). +- Cella scelta in-sample (volume-bars 24h, DONCH-10d): IS 1.45 → **HOLD −0.46** (2025 −7.4%, + 2026 −18.1%). Marginal NEUTRAL (corr 0.74 a TP01 = trend travestito), earns_slot_honest=False. +- Executability comunque assente: in alta attività le barre "24h" chiudono in 3.6-4.6h (p5) → il + cron orario le eseguirebbe in ritardo; i clock 4h richiedono monitoraggio sub-orario. + +**Lezione: campionare a tempo-informazione non normalizza nulla di monetizzabile sul feed +certificato; il clock non è la dimensione dove vive l'edge.** + +## Filone 2 — Calendario scadenze Deribit: FAIL (0/24 celle) + +`scripts/research/r0702_expiry_calendar.py`. Griglia dichiarata a priori: 4 finestre × 3 tipi expiry +(weekly ven 08:00 UTC / monthly ultimo ven / quarterly) × 2 asset, Bonferroni |t|≥3.08; 3 null +(placebo weekday, anchor-shift ±2/4h, permutation 500 calendari). + +- Nessuna cella passa Bonferroni (max |t| 2.69). Il post-expiry drift monthly (+0.61/+0.77%) ha il + segno della teoria ma: non batte il placebo giovedì, permutation pctl 84-95 (non estremo), si + INVERTE sul quarterly (dove l'OI massimo dovrebbe amplificarlo) ed è guidato dal 2019 → rumore. +- Unica cella robusta ai null: weekly [-24,0) negativa (gio→ven), ma è indistinguibile dal + day-of-week (famiglia SEA già morta) e netta fee fa Sharpe ~0.08 (52 eventi/anno × 0.10% RT). +- Best-in-sample tradabile: IS +0.90 → **HOLD −0.52**; DSR 0.230 su 24 trial. +- Nota tecnica: colpito ESATTAMENTE il pitfall pandas 2.x documentato (epoca in secondi da + `date_range` su tz-aware → matching vuoto); fix con epoca ms esplicita. La lezione 2026-07-01 + ha pagato. + +## Filone 3 — Anchor timing-luck di TP01 + tranching: FINDING (verificato dallo scettico) + +`scripts/research/r0702_tp01_offset.py` + scettico indipendente `r0702_skeptic_offset.py` +(ricostruzione bit-exact con codice diverso, zero riuso; h=0 riproduce `tp01_baseline_daily` +esattamente; vol-target ricalcolata per ancora; nessun look-ahead — verificato con troncamenti). + +**Claim 1 CONFERMATO: l'hold-out 2025-26 "Sharpe 0.31" di TP01 è in larga parte fortuna +dell'ancora 00:00 UTC.** La stessa strategia CANONICAL alle altre 23 ancore orarie: HOLD mediano +**0.04**, banda **[−0.13, +0.30]** — e il massimo è proprio h=0 (98° pctl; in-sample era al 10°: +sfortunato lì, fortunato qui). Spike, non plateau (h=23/h=1 fanno 0.13/0.04). Bootstrap dello +scettico: P(una qualsiasi ancora mostri uno spike ≥ così) = **0.86** — è il massimo atteso di 24 +stime correlate, non un'anomalia; best-anchor per finestra annuale ruota a caso (16→23→21→12→5→0, +Spearman ≈ 0, nessuna seasonality d'ancora). Formulazione corretta (NON "il vero hold-out è 0.07", +che ha CI95 [−1.2,+1.4] su 547 giorni): **l'hold-out 2025-26 non risolve l'edge di RITORNO di TP01** +(ritorno totale mediano ancore ≈ 0%); ciò che regge A OGNI ancora è il claim DIFENSIVO +(DD 6.7-10.1% vs ~60% B&H). Che è da sempre il vero valore dichiarato di TP01. + +**Claim 2 (tranching multi-ancora) RIDIMENSIONATO:** K=4 ≡ esattamente EW di 4 book ancorati +(diff 1.4e-17), turnover identico (~8.3/y), haircut $600 ≈ 0. MA il "miglioramento" IS 1.49→1.56 è +per ~90% *tornare alla media delle ancore* (h=0 era sfortunato in-sample): vs ancora tipica ++0.005 mediano, P(K4≤h0)=0.18 n.s.; il taglio DD strutturale è ~0.5pt (non 2.8: la mediana delle +singole è già 12.6%). Ciò che regge: la compressione della VARIANZA DELLA STIMA (HOLD +[−0.12,+0.30]→[−0.01,+0.13] a K=4) → **lens di reporting, non alpha**. + +**Impatto a valle (quantificato dallo scettico):** +- Blend SKH: HOLD 1.16 (h=0) → 0.97 mediana ancore, ma l'UPLIFT a mediana è +0.93 > +0.86 + dichiarato → **il verdetto ADDS di SKH01 era conservativo, regge**. +- Book 5-sleeve: HOLD 2.44 → 2.34 mediana (min 2.22), FULL 2.24→2.22 → eredita **~+0.10** di + fortuna d'ancora (non +0.25: il peso di TP01 diluisce). Nessuna decisione (GTAA/SKH/pesi) cambia. +- NON toccati: tutti i FULL (h=0 al 31° pctl = normale), plateau, deflated-Sharpe, fee-sweep, + causalità, validazione GTAA (equity OOS), gate weights_tilt_null (relativo). + +**Raccomandazione operativa (sottoscritta dallo scettico): cambio di narrativa e di standard di +reporting, NESSUN cambio del book live oggi.** (1) I numeri hold-out di TP01 si citano d'ora in poi +come banda d'ancora (mediana ~0.04, [−0.13,+0.30]) con valore = taglio DD; (2) valutazione +anchor-agnostic (media 24 ancore) come lens di reporting per i futuri hold-out; (3) NO deploy +K=2/K=4 a $600: guadagno ≈ 0, e a $225/asset di quota TP01 i delta per-ancora (~$1-2) sono sotto il +min-order $5 → degenererebbe comunque in K=1; inoltre le ancore intraday richiederebbero un feed +fresco fuori dal path certificato (staleness ×4 finestre di guasto). **Rivalutare K=2 (0,12) a +capitale ≥ ~5-10k.** (4) ⚠️ Audit analogo RACCOMANDATO su XS01 (rebalance 10g ancorato = spazio di +luck 10g×24h, potenzialmente peggiore) e SKH01 (fase griglia 230m/690m) — TP01 è oggi l'unico +sleeve de-luckato. + +## Filone 4 — Clock lenti + banded rebalancing: FAIL (negativo utile) + +`scripts/research/r0702_slow_clock.py`. Fatto a monte che chiude il filone: il fee drag di TP01 a +1d è **~0.40%/anno ≈ 0.03 Sharpe** (8 turnover/y × 0.10% RT) — il tetto di QUALSIASI risparmio. +- Clock N∈{2,3,5,7}g: degrado monotono dell'ensemble di fase (HOLD 0.34→0.11 da N=2 a N=7); lo + spread TRA fasi esplode a N≥5 (HOLD −0.19…+0.37 a N=7) = timing luck pura, coerente col filone 3. +- Bande di isteresi {2.5,5,10,20}%: tagliano il 77-94% degli ORDINI ma quasi zero TURNOVER (gli + ordini di TP01 sono micro-aggiustamenti del vol-target). Cella best-IS (band 20%): hold-out + 0.13 vs 0.30 baseline, e il "guadagno" IS è effetto-segnale (posizioni stantie fortunate), non + effetto-costo → fitting. Non-monotonia sulla griglia = firma di rumore. +- **Finding utile per il live: a $600 il vincolo min-order $5 È GIÀ la banda ottimale** (ordini + 427→111/y a costo Sharpe ~0, banda implicita 1.67%): cattura ~100% del risparmio catturabile. + Nessun cambio al book, a nessun capitale testato (600/2k/10k). + +## Filone 5 — Velocità del trend regime-condizionata: FAIL + +`scripts/research/r0702_regime_speed.py`. Pesi tra gli orizzonti TSMOM 30/90/180g condizionati al +percentile espandente di vol (realized e DVOL), 16 celle, sanity = riproduzione esatta del canonico. +- Cella best-IS (alta-vol→lento, linear, rv): FULL +0.06 ma **HOLD BTC −0.31**, 50/50 +0.005 → + dominanza fallita; multi-cut a segno instabile (+0.18/−0.08/+0.00). +- Null decisivo (300 pesi statici Dirichlet): la cella sta al pctl 0.71-0.72, sotto il p90 — + un peso statico casuale la batte spesso. Meccanismo smascherato: corr(peso-30d, Sharpe) = +0.93 → + le celle "regime" vincono in-sample perché tengono il tilt-30d la maggior parte dei giorni = + **tilt statico travestito da regime**, la stessa trappola di EW-STR. E il tilt statico verso il + 30d NON regge su BTC hold-out (−0.15). RV e DVOL indistinguibili come misura di regime (coerente + con l'esito overlay DVOL 2026-06-26). Pesi canonici 1/3-1/3-1/3 confermati. + +## Filoni 6-8 — CRT "Candle Range Theory" (base / multi-TF / contesto): FAIL 3/3 + +Il pattern sweep-and-reclaim a 3 candele (C1 range forte; C2 rompe un estremo ma chiude dentro = +"manipolazione"; C3 ingresso contro il breakout, stop dietro lo sweep, target all'estremo opposto), +meccanizzato onestamente in tre tagli. Overlap dichiarato: MRV01-11 e MIC07 non coprivano questa +meccanica; ora è coperta. + +**6. Base single-TF** (`r0702_crt_base.py`, 864 trial su 1h/4h/12h/1d, motore trade-level +conservativo SL-first cross-checkato con l'harness): **DSR 0.000** (il null best-of-grid atteso è +Sharpe 2.27, ottenuto 0.74); cella best-IS (4h long color-rule) IS 0.90 → HOLD 0.07 con 2026 a +WR 0%; anchor-shift +1/+2h flippa l'hold-out a −1.0 (artifact-risk); lo **short su sweep dell'alto +(la narrativa smart-money canonica) perde perfino in-sample (−0.59)**. Autopsia: l'expectancy IS +veniva dal time-exit in trend (beta del toro ETH 2021-23), non dal target strutturale. Dettaglio +informativo: CRT batte nettamente sia il fade incondizionato (IS −0.49) sia il breakout-confermato +(IS −0.31) → il close-back-inside FILTRA davvero tossicità, ma "meno tossico del fade morto" non è +un edge. + +**7. Multi-timeframe** (`r0702_crt_mtf.py`, 4h→15m e 1h→5m, ~10k trade): expectancy netta negativa +OVUNQUE (FULL e HOLD, entrambi gli asset); il MTF alza il R:R medio da ~3 a ~10-20 ma il WR collassa +da ~36% a 9-16% → "migliora" solo perdendo meno. **Refutazione strutturale: il ritest è informazione +negativa** — pattern CON ritest −40 bps, SENZA +52 bps (non tradabile: condiziona sul futuro): +aspettare il ritest per entrare seleziona sistematicamente i pattern peggiori; i buoni scappano +subito e il metodo non li prende mai. A fee ZERO l'edge lordo è ~0 (non è morte per fee: non c'è). +Stop 0.1-0.35% ineseguibili col cron orario (26-75% dei segnali già invalidati all'esecuzione). + +**8. Contesto** (`r0702_crt_context.py`, 22 trial: livelli prevday/Donchian/prevweek × equal-H/L × +FVG × sessioni): baseline incondizionata morta (8/8 celle HOLD<0); FVG semmai peggiora; sessione +EU consistentemente negativa; unica cella IS-positiva (sweep in Asia su livello prevday) uccisa 4 +volte (DSR 0.001, anchor-flip a −2/−4h con la firma di open_drive — la sessione "Asia" inizia dove +il livello prevday viene ricreato —, solo-ETH, morta a 0.20% fee). **Sottoprodotto prezioso: sugli +STESSI livelli prior-day, FOLLOW batte FADE ogni singolo anno 2019-2026** (follow IS/HOLD ++1.22/+1.25 vs fade −0.66/−1.46, corr −0.19/−0.27) → conferma indipendente e rafforzativa del lead +prevday-breakout in forward-monitor (paper_prevday). + +**Verdetto CRT: concetto Wyckoff riverniciato che, spogliato della discrezionalità, non contiene +edge su BTC/ETH Deribit certificati. La direzione giusta sui livelli affollati resta il FOLLOW +(breakout), non il fade — coerente con SKH01 (breakout) vivo e mean-reversion morta.** + +--- + +## Lezioni codificabili + +1. **Il timing-luck dell'ancora è una dimensione di multiple-testing NON coperta dai gate + esistenti** (deflated-Sharpe conta i trial, non le ancore implicite). Nuova regola di reporting: + ogni metrica hold-out di una strategia a ribilanciamento ancorato si cita con la banda d'ancora + (o la media anchor-agnostic). Candidato gate futuro: `anchor_luck_band()` in altlib. Audit + pendente: XS01 (10g), SKH01 (fase 230m/690m). +2. **La famiglia day_boundary_robust / anchor-shift continua a uccidere** (CRT base, CRT-Asia, + 6/24 celle expiry): ogni effetto legato a etichettatura di barre VA spostato d'ancora prima di + crederci. Confermata la regola 2026-06-21. +3. **Il fee drag di TP01 (~0.4%/anno) non è un problema da risolvere** — chiude a priori il filone + "esecuzione più furba" a questo turnover; il min-order small-cap è già la banda ottimale. +4. Il pitfall pandas tz-aware/epoca (2026-07-01) è stato evitato/gestito 2 volte su 9 agenti grazie + alla documentazione in CLAUDE.md — il costo di documentare i bug paga. + +## Stato finale + +- 0 nuovi sleeve; portafoglio e book live INVARIATI (5-sleeve 33/15/12/20/20; book Deribit + TP01+SKH01 75/25, flat da armamento — il segnale resta risk-off). +- Narrativa TP01 aggiornata (CLAUDE.md + docstring): hold-out = banda d'ancora, valore = difesa DD. +- Script: `scripts/research/r0702_{eventclock,expiry_calendar,tp01_offset,slow_clock,regime_speed, + crt_base,crt_mtf,crt_context,skeptic_offset}.py`. Test suite: invariata, verde. diff --git a/scripts/research/r0702_crt_base.py b/scripts/research/r0702_crt_base.py new file mode 100644 index 0000000..3b8aa80 --- /dev/null +++ b/scripts/research/r0702_crt_base.py @@ -0,0 +1,507 @@ +"""r0702_crt_base — CRT "Candle Range Theory", versione BASE single-TF (pattern meccanizzato). + +FILONE 2026-07-02. Falso breakout codificato in 3 candele (turtle soup / spring-upthrust): + C1 (range): candela direzionale forte -> body/range >= b AND range >= k*ATR14 (griglia b,k) + C2 (manipolazione): rompe un estremo di C1 di almeno s*ATR14 (griglia s) ma CHIUDE DENTRO + il range di C1. Flag colore opzionale (short: C2 rossa che apre sopra il close di C1; + long: C2 verde che apre sotto il close di C1). + C3 (ingresso): entry a open C3 = close C2 (deciso con dati <= close C2 -> causale). + SL = estremo di C2 (punto dello sweep). TP = estremo OPPOSTO del range di C1. + Filtro R:R >= 1.3 a entry. Direzioni: short su sweep dell'alto, long su sweep del basso. + +OVERLAP DICHIARATO con la ricerca esistente (grep dei docstring runs/MRV*.py + MIC07): + - MRV01-11 = mean-reversion su INDICATORI (RSI2, BB, z-score, IBS, W%R, consec-down, + gap-fill, CCI, stochastic, VWAP-dev, %b) — nessuna testa il pattern 3-candele + sweep+close-back-inside con SL/TP strutturali. La famiglia MR generica e' MORTA sul + feed certificato: CRT e' una MR *condizionata da un evento di liquidita'*, quindi il + prior e' fortemente negativo — serve battere il null del fade incondizionato. + - MIC07 (pin-bar rejection al supporto) e' il parente piu' vicino: rejection candle + single-bar a un N-bar low. CRT differisce: riferimento = range di C1 forte (1 barra), + sweep quantificato in ATR, close-back-inside esplicito, TP strutturale (estremo opposto + di C1) e non R-multiple. Overlap concettuale parziale, meccanica diversa. + +GATES: selezione cella SOLO in-sample pre-2025; deflated Sharpe su TUTTI i trial +(cella x tf x direzione); ANCHOR-SHIFT (+1/+2/+4h) sul resample 4h/12h/1d; fee sweep +0.00-0.20% RT; marginal_vs_tp01 se Sharpe standalone >= 0.5. +NULL decisivi: (i) fade INCONDIZIONATO dello stesso estremo (senza close-back-inside); +(ii) condizione INVERTITA (C2 chiude FUORI = breakout confermato, trade col breakout). + +Motore trade-level CONSERVATIVO (specchia src/backtest/harness.backtest_signals): +entry a close[i]; exit scan da i+1; SL/TP fillati AL LIVELLO su high/low; se nella stessa +barra sono toccati entrambi scatta PRIMA lo STOP (worst-case); time-exit a close dopo +max_hold barre; nessun overlap (una posizione alla volta per asset). Fee 0.10% RT. +Equity mark-to-market per barra (lente Sharpe daily-compounded, convenzione progetto). + +Run: uv run python scripts/research/r0702_crt_base.py +""" +from __future__ import annotations + +import sys +import time +from itertools import product + +sys.path.insert(0, "/opt/docker/PythagorasGoal/scripts/research/alt") +import altlib as al # noqa: E402 + +import numpy as np # noqa: E402 +import pandas as pd # noqa: E402 + +HOLDOUT = al.HOLDOUT # 2025-01-01 UTC +FEE_RT = 2 * al.FEE_SIDE # 0.10% round-trip +RR_MIN = 1.3 +TFS = ("1h", "4h", "12h", "1d") +RULES = {"4h": "4h", "12h": "12h", "1d": "1D"} +ASSETS = al.CERTIFIED # BTC, ETH +MIN_IS_TRADES = 25 # trade combinati minimi in-sample per cella eleggibile + +GRID = [dict(b=b, k=k, s=s, color=col, mh=mh) + for b, k, s, col, mh in product((0.5, 0.7), (1.0, 1.5), + (0.0, 0.1, 0.25), (False, True), (5, 10, 20))] +DIRS = ("long", "short", "both") + + +# =========================================================================== +# DATI (anchor 00:00 UTC di default; anchor spostabile per il gate anchor-shift) +# =========================================================================== +def resample_anchor(df_1h: pd.DataFrame, rule: str, offset_hours: int) -> pd.DataFrame: + """Come trend_portfolio.resample_tf (label/closed='left') ma con ancora spostata di + +offset_hours. Niente .view('int64'): epoca esplicita via // Timedelta (tz-aware safe).""" + g = df_1h.copy() + idx = pd.to_datetime(g["timestamp"], unit="ms", utc=True) + idx.name = "dt" + g.index = idx + out = g.resample(rule, label="left", closed="left", + offset=pd.Timedelta(hours=offset_hours)).agg( + {"open": "first", "high": "max", "low": "min", "close": "last", "volume": "sum"}) + out = out.dropna(subset=["open"]) + out["datetime"] = out.index + epoch = pd.Timestamp("1970-01-01", tz="UTC") + out["timestamp"] = ((out.index - epoch) // pd.Timedelta(milliseconds=1)).astype("int64") + return out.reset_index(drop=True)[["timestamp", "open", "high", "low", "close", + "volume", "datetime"]] + + +_PREP_CACHE: dict = {} + + +def prep(asset: str, tf: str, anchor: int = 0) -> dict: + key = (asset, tf, anchor) + if key in _PREP_CACHE: + return _PREP_CACHE[key] + if anchor == 0 or tf == "1h": + df = al.get(asset, tf) + else: + df = resample_anchor(al.get(asset, "1h"), RULES[tf], anchor) + d = dict( + df=df, + o=df["open"].values.astype(float), h=df["high"].values.astype(float), + l=df["low"].values.astype(float), c=df["close"].values.astype(float), + atr=al.atr(df, 14), + idx=pd.DatetimeIndex(pd.to_datetime(df["datetime"], utc=True)), + ) + _PREP_CACHE[key] = d + return d + + +# =========================================================================== +# DETECTION (vettoriale, causale: tutto deciso con OHLC fino alla barra i = C2; +# l'ATR usato e' quello di C1 (i-1) -> ancora piu' conservativo) +# =========================================================================== +def _shift1(x: np.ndarray) -> np.ndarray: + out = np.empty_like(x); out[0] = np.nan; out[1:] = x[:-1] + return out + + +def detect(d: dict, b: float, k: float, s: float, color: bool, variant: str) -> dict: + """Ritorna {dir: (indici C2, sl, tp)} per variant in {'crt','fade','breakout'}. + Indice i = barra C2; C1 = i-1. Entry (gestita dal motore) = close[i].""" + o, h, l, c = d["o"], d["h"], d["l"], d["c"] + h1, l1, o1, c1 = _shift1(h), _shift1(l), _shift1(o), _shift1(c) + atr1 = _shift1(d["atr"]) + rng1 = h1 - l1 + body1 = np.abs(c1 - o1) + with np.errstate(invalid="ignore", divide="ignore"): + strong = (np.isfinite(atr1) & (atr1 > 0) & (rng1 > 0) + & (body1 / rng1 >= b) & (rng1 >= k * atr1)) + sweep_up = strong & (h > h1 + s * atr1) # C2 rompe l'alto di C1 + sweep_dn = strong & (l < l1 - s * atr1) # C2 rompe il basso di C1 + out = {} + if variant == "crt": + sh = sweep_up & (c < h1) & (c > l1) # chiude DENTRO il range di C1 + lg = sweep_dn & (c > l1) & (c < h1) + if color: + sh &= (c < o) & (o > c1) # rossa che apre sopra il close di C1 + lg &= (c > o) & (o < c1) # verde che apre sotto il close di C1 + sh &= (h > c) & ((c - l1) / np.where(h - c > 0, h - c, np.nan) >= RR_MIN) + lg &= (c > l) & ((h1 - c) / np.where(c - l > 0, c - l, np.nan) >= RR_MIN) + out["short"] = (np.where(sh)[0], h, l1) # SL=high C2, TP=low C1 + out["long"] = (np.where(lg)[0], l, h1) # SL=low C2, TP=high C1 + elif variant == "fade": + # NULL (i): stesso sweep, NESSUNA richiesta di close-back-inside (no colore). + # Solo validita' geometrica (TP dal lato giusto dell'entry). + sh = sweep_up & (c > l1) + lg = sweep_dn & (c < h1) + sh &= (h > c) & ((c - l1) / np.where(h - c > 0, h - c, np.nan) >= RR_MIN) + lg &= (c > l) & ((h1 - c) / np.where(c - l > 0, c - l, np.nan) >= RR_MIN) + out["short"] = (np.where(sh)[0], h, l1) + out["long"] = (np.where(lg)[0], l, h1) + elif variant == "breakout": + # NULL (ii): condizione INVERTITA — C2 chiude FUORI dal range di C1 = + # breakout confermato, trade IN DIREZIONE del breakout. + # SL = livello rotto (rientro nel range = fallimento), TP = measured move + # (range di C1 proiettato oltre il livello). Stesso filtro R:R. + lg = sweep_up & (c > h1) # rompe l'alto e chiude sopra -> LONG + sh = sweep_dn & (c < l1) # rompe il basso e chiude sotto -> SHORT + tp_lg = h1 + rng1 + tp_sh = l1 - rng1 + lg &= (tp_lg > c) & ((tp_lg - c) / np.where(c - h1 > 0, c - h1, np.nan) >= RR_MIN) + sh &= (c > tp_sh) & ((c - tp_sh) / np.where(l1 - c > 0, l1 - c, np.nan) >= RR_MIN) + out["long"] = (np.where(lg)[0], h1, tp_lg) # SL=high C1, TP=measured move + out["short"] = (np.where(sh)[0], l1, tp_sh) # SL=low C1 + else: + raise ValueError(variant) + return out + + +def merge_dirs(sig: dict, which: str): + """Lista ordinata di (i, dir, sl, tp) per direzione 'long'/'short'/'both'.""" + rows = [] + if which in ("long", "both"): + ii, sl, tp = sig["long"] + rows += [(int(i), 1, float(sl[i]), float(tp[i])) for i in ii] + if which in ("short", "both"): + ii, sl, tp = sig["short"] + rows += [(int(i), -1, float(sl[i]), float(tp[i])) for i in ii] + rows.sort(key=lambda r: r[0]) + return rows + + +# =========================================================================== +# MOTORE TRADE-LEVEL (conservativo; specchia backtest_signals: SL prioritario) +# =========================================================================== +def run_trades(d: dict, rows: list, mh: int, fee_rt: float = FEE_RT): + """Ritorna (trades, barnet). trades: (i_entry, i_exit, dir, net, R, gross). + barnet: rendimento netto per-barra mark-to-market (fee 50/50 su entry/exit bar).""" + c, h, l = d["c"], d["h"], d["l"] + n = len(c) + barnet = np.zeros(n) + trades = [] + busy_until = -1 + for i, dr, sl, tp in rows: + if i <= busy_until or i >= n - 1: + continue + entry = c[i] + exit_idx = min(i + mh, n - 1) + exit_price = c[exit_idx] + for j in range(i + 1, min(i + mh, n - 1) + 1): + if dr == 1: + if l[j] <= sl: # STOP prima (worst-case) + exit_price, exit_idx = sl, j; break + if h[j] >= tp: + exit_price, exit_idx = tp, j; break + else: + if h[j] >= sl: + exit_price, exit_idx = sl, j; break + if l[j] <= tp: + exit_price, exit_idx = tp, j; break + exit_price, exit_idx = c[j], j + gross = dr * (exit_price / entry - 1.0) + net = gross - fee_rt + risk = abs(sl - entry) / entry + R = net / risk if risk > 0 else np.nan + trades.append((i, exit_idx, dr, net, R, gross)) + pp = entry + for j in range(i + 1, exit_idx + 1): + pj = exit_price if j == exit_idx else c[j] + barnet[j] += dr * (pj / pp - 1.0) + pp = pj + barnet[i] -= fee_rt / 2 + barnet[exit_idx] -= fee_rt / 2 + busy_until = exit_idx + return trades, barnet + + +def daily_series(d: dict, barnet: np.ndarray) -> pd.Series: + return al._to_daily(pd.Series(barnet, index=d["idx"])) + + +def combo_daily(dailies: dict) -> pd.Series: + J = pd.concat(dailies, axis=1, join="inner").fillna(0.0) + return 0.5 * J[ASSETS[0]] + 0.5 * J[ASSETS[1]] + + +def series_metrics(daily: pd.Series) -> dict: + def _dd(s): + eq = np.cumprod(1.0 + s.values); pk = np.maximum.accumulate(eq) + return float(np.max((pk - eq) / pk)) if len(eq) else 0.0 + ins, hold = daily[daily.index < HOLDOUT], daily[daily.index >= HOLDOUT] + yearly = {int(y): round(float(np.prod(1 + g.values) - 1), 4) + for y, g in daily.groupby(daily.index.year)} + return dict(full_sh=round(al._sh(daily), 3), is_sh=round(al._sh(ins), 3), + hold_sh=round(al._sh(hold), 3), full_dd=round(_dd(daily), 4), + hold_dd=round(_dd(hold), 4), full_ret=round(float(np.prod(1 + daily.values) - 1), 4), + hold_ret=round(float(np.prod(1 + hold.values) - 1), 4), yearly=yearly) + + +def trade_stats(trades: list, idx: pd.DatetimeIndex) -> dict: + if not trades: + return dict(n=0, n_is=0, n_hold=0, wr=None, avg_R=None, exp_net=None) + t_entry = idx[[t[0] for t in trades]] + net = np.array([t[3] for t in trades]); R = np.array([t[4] for t in trades]) + is_m = np.asarray(t_entry < HOLDOUT) + + def _blk(m): + if m.sum() == 0: + return dict(n=0, wr=None, avg_R=None, exp_net=None) + return dict(n=int(m.sum()), wr=round(float(np.mean(net[m] > 0) * 100), 1), + avg_R=round(float(np.nanmean(R[m])), 3), + exp_net=round(float(np.mean(net[m]) * 100), 3)) + full = _blk(np.ones(len(net), bool)) + per_year = {} + for y in sorted(set(t_entry.year)): + per_year[int(y)] = _blk(np.asarray(t_entry.year == y)) + return dict(n=full["n"], n_is=int(is_m.sum()), n_hold=int((~is_m).sum()), + wr=full["wr"], avg_R=full["avg_R"], exp_net=full["exp_net"], + is_blk=_blk(is_m), hold_blk=_blk(~is_m), per_year=per_year) + + +# =========================================================================== +# RUNNER di un trial (cella x tf x direzione) su entrambi gli asset +# =========================================================================== +def run_trial(tf: str, p: dict, which: str, variant: str = "crt", + fee_rt: float = FEE_RT, anchor: int = 0): + dailies, all_trades, all_stats = {}, {}, {} + for a in ASSETS: + d = prep(a, tf, anchor) + sig = detect(d, p["b"], p["k"], p["s"], p["color"], variant) + rows = merge_dirs(sig, which) + trades, barnet = run_trades(d, rows, p["mh"], fee_rt) + dailies[a] = daily_series(d, barnet) + all_trades[a] = trades + all_stats[a] = trade_stats(trades, d["idx"]) + daily = combo_daily(dailies) + sm = series_metrics(daily) + n_is = sum(st["n_is"] for st in all_stats.values()) + n_full = sum(st["n"] for st in all_stats.values()) + return dict(tf=tf, params=p, dir=which, variant=variant, daily=daily, + metrics=sm, per_asset_stats=all_stats, n_is=n_is, n_full=n_full) + + +def pooled_trade_stats(trial: dict) -> dict: + """Statistiche trade POOLED sui due asset (per il report della cella scelta).""" + trades, idxs = [], [] + for a in ASSETS: + d = prep(a, trial["tf"]) + for t in trial["_raw_trades"][a]: + trades.append(t); idxs.append(d["idx"][t[0]]) + if not trades: + return dict(n=0) + order = np.argsort(np.array([i.value for i in idxs])) + trades = [trades[k] for k in order] + idx = pd.DatetimeIndex([idxs[k] for k in order]) + return _pooled(trades, idx) + + +def _pooled(trades, idx): + net = np.array([t[3] for t in trades]); R = np.array([t[4] for t in trades]) + is_m = np.asarray(idx < HOLDOUT) + + def _blk(m): + if m.sum() == 0: + return dict(n=0, wr=None, avg_R=None, exp_net=None) + return dict(n=int(m.sum()), wr=round(float(np.mean(net[m] > 0) * 100), 1), + avg_R=round(float(np.nanmean(R[m])), 3), + exp_net=round(float(np.mean(net[m]) * 100), 3)) + out = dict(full=_blk(np.ones(len(net), bool)), is_blk=_blk(is_m), hold_blk=_blk(~is_m), + per_year={int(y): _blk(np.asarray(idx.year == y)) for y in sorted(set(idx.year))}) + return out + + +# =========================================================================== +# MAIN +# =========================================================================== +def main(): + t0 = time.time() + print("=" * 96) + print("r0702 CRT — Candle Range Theory BASE single-TF | fee 0.10% RT | hold-out 2025-01-01+") + print("Griglia: b(0.5,0.7) x k(1.0,1.5) x s(0.0,0.1,0.25) x color(off,on) x max_hold(5,10,20)") + print(f"= {len(GRID)} celle x {len(TFS)} TF x {len(DIRS)} direzioni = " + f"{len(GRID) * len(TFS) * len(DIRS)} trial (tutti contati nel DSR)") + print("=" * 96) + + for a in ASSETS: + d = prep(a, "1d") + print(f" dati {a} 1d: {d['idx'][0].date()} -> {d['idx'][-1].date()} ({len(d['c'])} barre)") + + # ---- 1) griglia completa (righe leggere; il daily si ricalcola per la scelta) ---- + rows = [] + freq_by_tf = {tf: [] for tf in TFS} + for tf in TFS: + years = {} + for a in ASSETS: + d = prep(a, tf) + years[a] = (d["idx"][-1] - d["idx"][0]).total_seconds() / 86400 / 365.25 + span_y = float(np.mean(list(years.values()))) + for p in GRID: + # detection condivisa fra direzioni e mh (mh influenza solo il motore) + for which in DIRS: + tr = run_trial(tf, p, which) + m = tr["metrics"] + rows.append(dict(tf=tf, **p, dir=which, is_sh=m["is_sh"], full_sh=m["full_sh"], + hold_sh=m["hold_sh"], n_is=tr["n_is"], n_full=tr["n_full"])) + if which == "both": + freq_by_tf[tf].append(tr["n_full"] / (2 * span_y)) # trade/anno per asset + print(f" [grid] tf={tf} fatto ({time.time() - t0:.0f}s)") + + R = pd.DataFrame(rows) + print("\n--- FREQUENZA PATTERN (CRT, entrambe le direzioni, trade/anno PER ASSET) ---") + for tf in TFS: + f = np.array(freq_by_tf[tf]) + print(f" {tf:>4s}: mediana {np.median(f):6.1f} min {f.min():6.1f} max {f.max():6.1f} " + f"(su {len(f)} celle)") + + # ---- 2) selezione cella SOLO in-sample (pre-2025) ---- + elig = R[(R.n_is >= MIN_IS_TRADES) & np.isfinite(R.is_sh)].copy() + print(f"\n--- SELEZIONE IN-SAMPLE: {len(elig)}/{len(R)} trial eleggibili " + f"(>= {MIN_IS_TRADES} trade IS combinati) ---") + top = elig.sort_values("is_sh", ascending=False).head(12) + cols = ["tf", "b", "k", "s", "color", "mh", "dir", "is_sh", "hold_sh", "full_sh", "n_is", "n_full"] + print(top[cols].to_string(index=False)) + + if len(elig) == 0: + print("\nVERDETTO: FAIL — nessuna cella con abbastanza trade in-sample.") + return + + best = elig.sort_values("is_sh", ascending=False).iloc[0] + p = dict(b=float(best.b), k=float(best.k), s=float(best.s), + color=bool(best.color), mh=int(best.mh)) + tf, which = str(best.tf), str(best.dir) + print(f"\n=== CELLA SCELTA (max Sharpe IN-SAMPLE, hold-out solo riportato) ===") + print(f" tf={tf} dir={which} {p}") + + # ricalcolo completo della cella scelta (con trade grezzi per il pooled report) + chosen = run_trial(tf, p, which) + chosen["_raw_trades"] = {} + for a in ASSETS: + d = prep(a, tf) + sig = detect(d, p["b"], p["k"], p["s"], p["color"], "crt") + trades, _ = run_trades(d, merge_dirs(sig, which), p["mh"]) + chosen["_raw_trades"][a] = trades + m = chosen["metrics"] + print(f" COMBINED 50/50: FULL Sh {m['full_sh']} IS Sh {m['is_sh']} HOLD Sh {m['hold_sh']} " + f"| FULL ret {m['full_ret'] * 100:+.1f}% DD {m['full_dd'] * 100:.1f}% " + f"| HOLD ret {m['hold_ret'] * 100:+.1f}% DD {m['hold_dd'] * 100:.1f}%") + print(f" per-anno (ret combo): " + " ".join(f"{y}:{v * 100:+.1f}%" for y, v in m["yearly"].items())) + ps = pooled_trade_stats(chosen) + if ps.get("full", {}).get("n", 0) > 0: + f_, i_, h_ = ps["full"], ps["is_blk"], ps["hold_blk"] + print(f" trade POOLED: n={f_['n']} WR={f_['wr']}% avgR={f_['avg_R']} exp={f_['exp_net']}%" + f" | IS n={i_['n']} WR={i_['wr']}% avgR={i_['avg_R']} exp={i_['exp_net']}%" + f" | HOLD n={h_['n']} WR={h_['wr']}% avgR={h_['avg_R']} exp={h_['exp_net']}%") + print(" trade per anno: " + " ".join( + f"{y}:n{b['n']}/wr{b['wr']}/exp{b['exp_net']}%" for y, b in ps["per_year"].items())) + for a in ASSETS: + st = chosen["per_asset_stats"][a] + print(f" {a}: n={st['n']} (IS {st['n_is']}/HOLD {st['n_hold']}) WR={st['wr']}% " + f"avgR={st['avg_R']} exp={st['exp_net']}%") + + # per-direzione della cella scelta (stessi parametri) + print("\n--- CELLA SCELTA per DIREZIONE (stessi parametri) ---") + for wdir in DIRS: + tr = run_trial(tf, p, wdir) + mm = tr["metrics"] + print(f" {wdir:>5s}: IS {mm['is_sh']:+.2f} HOLD {mm['hold_sh']:+.2f} FULL {mm['full_sh']:+.2f} " + f"n={tr['n_full']} (IS {tr['n_is']})") + + # ---- sanity cross-check vs harness ufficiale (al.eval_signals) ---- + print("\n--- CROSS-CHECK vs al.eval_signals (harness ufficiale, stessa convenzione) ---") + for a in ASSETS: + d = prep(a, tf) + sig = detect(d, p["b"], p["k"], p["s"], p["color"], "crt") + entries = [None] * len(d["c"]) + for i, dr, sl, tp in merge_dirs(sig, which): + entries[i] = dict(dir=dr, sl=sl, tp=tp, max_bars=p["mh"]) + ev = al.eval_signals(d["df"], entries, fee_rt=FEE_RT, asset=a, tf=tf) + mine = chosen["_raw_trades"][a] + my_ret = float(np.prod([1 + t[3] for t in mine]) - 1) + print(f" {a}: harness n={ev['n_trades']} ret={ev['full']['ret'] * 100:+.1f}% " + f"| mio n={len(mine)} ret={my_ret * 100:+.1f}% " + f"{'OK' if ev['n_trades'] == len(mine) else 'MISMATCH!'}") + + # ---- 3) DSR su TUTTI i trial ---- + all_sr = [r["full_sh"] for r in rows if np.isfinite(r["full_sh"]) and r["n_full"] >= 1] + dsr, sr0 = al.deflated_sharpe(m["full_sh"], all_sr, chosen["daily"]) + print(f"\n--- DEFLATED SHARPE: DSR={dsr:.3f} (PASS>=0.95) expected-null-max Sh={sr0:.2f} " + f"trial contati={len(all_sr)} (su {len(rows)} totali; esclusi 0-trade) ---") + + # ---- 4) ANCHOR-SHIFT (+1/+2/+4h) ---- + print("\n--- ANCHOR-SHIFT (ancora resample spostata; pattern vero ~invariante) ---") + anchor_rows = {} + if tf == "1h": + print(" tf=1h nativo: nessuna dipendenza dall'ancora del resample (N/A).") + alt = elig[elig.tf != "1h"].sort_values("is_sh", ascending=False) + if len(alt): + b2 = alt.iloc[0] + p2 = dict(b=float(b2.b), k=float(b2.k), s=float(b2.s), color=bool(b2.color), mh=int(b2.mh)) + print(f" (test eseguito sulla miglior cella IS a tf>=4h: tf={b2.tf} dir={b2.dir} {p2})") + for off in (0, 1, 2, 4): + tr = run_trial(str(b2.tf), p2, str(b2.dir), anchor=off) + mm = tr["metrics"] + anchor_rows[off] = mm + print(f" anchor +{off}h: IS {mm['is_sh']:+.2f} HOLD {mm['hold_sh']:+.2f} " + f"FULL {mm['full_sh']:+.2f} n={tr['n_full']}") + else: + for off in (0, 1, 2, 4): + tr = run_trial(tf, p, which, anchor=off) + mm = tr["metrics"] + anchor_rows[off] = mm + print(f" anchor +{off}h: IS {mm['is_sh']:+.2f} HOLD {mm['hold_sh']:+.2f} " + f"FULL {mm['full_sh']:+.2f} n={tr['n_full']}") + if anchor_rows: + fulls = [v["full_sh"] for v in anchor_rows.values()] + flip = (max(fulls) > 0) and (min(fulls) < 0) + print(f" spread FULL Sh = {max(fulls) - min(fulls):+.2f} " + f"{'SIGN-FLIP -> ARTIFACT-RISK' if flip else 'nessun sign-flip'}") + + # ---- 5) FEE SWEEP 0.00-0.20% RT ---- + print("\n--- FEE SWEEP (cella scelta) ---") + for fr in (0.0, 0.0005, 0.001, 0.0015, 0.002): + tr = run_trial(tf, p, which, fee_rt=fr) + mm = tr["metrics"] + print(f" fee {fr * 100:.2f}%RT: IS {mm['is_sh']:+.2f} HOLD {mm['hold_sh']:+.2f} " + f"FULL {mm['full_sh']:+.2f}") + + # ---- 6) NULL DECISIVI ---- + print("\n--- NULL (i): FADE INCONDIZIONATO dello stesso estremo (no close-back-inside) ---") + p_null = dict(p, color=False) + for var, lbl in (("fade", "fade-incond"), ("breakout", "breakout-conf")): + tr = run_trial(tf, p_null, which, variant=var) + mm = tr["metrics"] + print(f" {lbl:>14s}: IS {mm['is_sh']:+.2f} HOLD {mm['hold_sh']:+.2f} FULL {mm['full_sh']:+.2f} " + f"n={tr['n_full']} (IS {tr['n_is']}) per-anno " + + " ".join(f"{y}:{v * 100:+.0f}%" for y, v in mm["yearly"].items())) + tr = run_trial(tf, p, which, variant="crt") + mm = tr["metrics"] + print(f" {'CRT (rif.)':>14s}: IS {mm['is_sh']:+.2f} HOLD {mm['hold_sh']:+.2f} FULL {mm['full_sh']:+.2f} " + f"n={tr['n_full']} (IS {tr['n_is']})") + + # ---- 7) MARGINAL vs TP01 (solo se standalone >= 0.5) ---- + if max(m["full_sh"], m["is_sh"]) >= 0.5: + print("\n--- MARGINAL vs TP01 (standalone >= 0.5) ---") + marg = al.marginal_vs_tp01(chosen["daily"]) + keys = ("marginal_verdict", "corr_full", "corr_hold", "cand_insample_sharpe", + "has_insample_edge", "is_hedge", "multicut_uplift", "multicut_persistent", + "robust_oos", "beta_to_tp01", "resid_sharpe_full") + for kk in keys: + print(f" {kk}: {marg.get(kk)}") + for w, dd in marg.get("blends", {}).items(): + print(f" blend {w}: full {dd['full']} (uplift {dd['uplift_full']:+.3f}) " + f"hold {dd['hold']} (uplift {dd['uplift_hold']})") + else: + print(f"\n--- MARGINAL vs TP01: SALTATO (standalone FULL {m['full_sh']} / IS {m['is_sh']} < 0.5) ---") + + print(f"\n[done in {time.time() - t0:.0f}s]") + + +if __name__ == "__main__": + main() diff --git a/scripts/research/r0702_crt_context.py b/scripts/research/r0702_crt_context.py new file mode 100644 index 0000000..e270c00 --- /dev/null +++ b/scripts/research/r0702_crt_context.py @@ -0,0 +1,505 @@ +"""r0702_crt_context.py — CRT CON CONTESTO (2026-07-02). + +FILONE: la scuola "Candle Range Theory" dice che lo sweep-and-reclaim vale SOLO su zone +importanti (liquidita' sopra massimi/minimi rilevanti, FVG, sessione giusta). Qui testiamo se +i FILTRI DI CONTESTO trasformano un fade (gia' morto in versione generica sul feed certificato) +in un edge. ONESTA' PRIMA DI TUTTO. + +SETUP BASE FISSO (identico per tutte le celle, deciso A PRIORI prima di guardare i numeri): + * TF in {1h, 4h} (4h = resample leak-free da 1h via altlib.get). + * C2 = barra che SUPERA un livello di riferimento e CHIUDE dal lato opposto: + SHORT: high[i] > lvl_hi AND close[i] < lvl_hi (sweep del massimo + reclaim) + LONG : low[i] < lvl_lo AND close[i] > lvl_lo (sweep del minimo + reclaim) + (barra che sweppa ENTRAMBI i livelli e chiude in mezzo = ambigua -> scartata, dichiarato) + * Entry a close[i] (decisione con dati <= close[i], eseguibile). + * Stop DIETRO l'estremo di C2: estremo +/- 0.10*ATR14 (causale). + * Target = R FISSO 1.5:1 (scelto a priori; NON centro-range: il centro di un Donchian in + trend e' asimmetrico/ambiguo, R fisso e' uniforme su tutti i level-type). + * max_hold 20 barre; fill conservativi (SL prioritario se TP e SL nella stessa barra, + identico all'harness src/backtest/harness.backtest_signals); fee 0.10% RT. + +LIVELLI (tutti causali, shift(1) su aggregati di periodi COMPLETI precedenti): + prevday = high/low del giorno UTC precedente (barre open-labeled, groupby giorno -> shift) + don20/55 = max/min delle N barre STRETTAMENTE precedenti (al.donchian, shift(1) built-in) + prevweek = high/low della settimana ISO precedente (lunedi' 00 UTC) + +⚠️ OVERLAP DICHIARATO col lead PREVDAY-BREAKOUT in forward-monitor +(src/strategies/prevday_breakout.py + scripts/live/paper_prevday.py): STESSI livelli prior-day, +condizionamento OPPOSTO — il lead SEGUE il break decisivo (close > lvl + 0.30*range), qui si +FADEA il reclaim (close torna dentro). Se entrambi avessero edge sugli stessi livelli, uno dei +due e' rumore -> confronto esplicito fade-vs-follow (corr daily + chi vince dove) in fondo. + +FILTRI DI CONTESTO (la parte "discrezionale" della CRT, meccanizzata): + EQ = equal highs/lows: il livello e' stato toccato >=2 volte entro 0.10*ATR14 nelle ultime + N barre (N = lookback del livello stesso; prevday=2 giorni, prevweek=7 giorni). + FVG = esiste un fair-value-gap a 3 candele NON ancora riempito nelle ultime 20 barre, nella + direzione del trade (short: FVG bullish sotto il prezzo non riempito = magnete giu'; + long: FVG bearish sopra non riempito). Meccanizzazione SEMPLICE di un concetto fuzzy + discrezionale — limiti dichiarati: k=20 fisso, zona "non riempita" = mai traversata + interamente, nessuna nozione di "displacement" o "premium/discount". + SES = sessione dello sweep (ora UTC di apertura barra): Asia 00-08 / Europa 08-14 / US 14-22. + Solo a 1h (a 4h la label di sessione e' troppo grossolana). ⚠️ OGNI cella sessione + passa un anchor-shift +/-2/4h (analogo di al.day_boundary_robust a livello trade) + prima di essere creduta. + +GATES: selezione SOLO in-sample pre-2025 (HOLDOUT altlib = 2025-01-01); hold-out riportato mai +usato per scegliere; al.deflated_sharpe su TUTTI i 22 trial; fee sweep 0.00-0.20% RT; se il +best-IS regge (Sharpe >= 0.5) -> al.marginal_vs_tp01. Causalita': livelli ricalcolati su +prefisso troncato e confrontati (check esplicito in fondo). Niente .view("int64"), niente +ffill mixed-TF. + +Uso: uv run python scripts/research/r0702_crt_context.py +""" +from __future__ import annotations + +import sys +from pathlib import Path + +sys.path.insert(0, "/opt/docker/PythagorasGoal/scripts/research/alt") +import altlib as al # noqa: E402 + +import numpy as np # noqa: E402 +import pandas as pd # noqa: E402 + +ROOT = Path("/opt/docker/PythagorasGoal") +if str(ROOT) not in sys.path: + sys.path.insert(0, str(ROOT)) +from src.backtest.harness import backtest_signals # noqa: E402 +from src.strategies.prevday_breakout import target as prevday_follow_target # noqa: E402 + +HOLDOUT = al.HOLDOUT +FEE_RT = 0.001 # 0.10% round-trip (Deribit taker) +MAX_HOLD = 20 # barre +R_MULT = 1.5 # target R fisso 1.5:1 (a priori) +SL_ATR_BUF = 0.10 # stop = estremo C2 +/- 0.10*ATR14 (a priori) +EQ_TOL_ATR = 0.10 # tolleranza equal highs/lows +EQ_MIN_TOUCH = 2 +FVG_K = 20 # lookback barre per FVG non riempito +ASSETS = ("BTC", "ETH") +SESSIONS = {"asia": (0, 8), "eu": (8, 14), "us": (14, 22)} +LEVELS = ("prevday", "don20", "don55", "prevweek") + + +# =========================================================================== +# LIVELLI (causali) +# =========================================================================== +def prior_day_levels(df: pd.DataFrame, shift_h: int = 0): + """High/low del giorno UTC PRECEDENTE (shift(1) sul groupby giorno -> strettamente + prima di oggi). shift_h sposta il confine del giorno (per l'anchor-shift test).""" + dt = pd.to_datetime(df["datetime"], utc=True) + pd.Timedelta(hours=shift_h) + day = dt.dt.floor("1D") + g = pd.DataFrame({"day": day.values, + "high": df["high"].values.astype(float), + "low": df["low"].values.astype(float)}) + per = g.groupby("day").agg(dh=("high", "max"), dl=("low", "min")) + m = pd.DataFrame({"dh": per["dh"].shift(1), "dl": per["dl"].shift(1)}).reindex(g["day"].values) + return m["dh"].values, m["dl"].values + + +def prior_week_levels(df: pd.DataFrame): + """High/low della settimana ISO PRECEDENTE (lunedi' 00 UTC, shift(1)).""" + dt = pd.to_datetime(df["datetime"], utc=True) + day = dt.dt.floor("1D") + week = (day - pd.to_timedelta(dt.dt.dayofweek, unit="D")) + g = pd.DataFrame({"wk": week.values, + "high": df["high"].values.astype(float), + "low": df["low"].values.astype(float)}) + per = g.groupby("wk").agg(wh=("high", "max"), wl=("low", "min")) + m = pd.DataFrame({"wh": per["wh"].shift(1), "wl": per["wl"].shift(1)}).reindex(g["wk"].values) + return m["wh"].values, m["wl"].values + + +def get_levels(df: pd.DataFrame, level: str, shift_h: int = 0): + if level == "prevday": + return prior_day_levels(df, shift_h) + if level == "prevweek": + return prior_week_levels(df) + if level == "don20": + return al.donchian(df, 20) + if level == "don55": + return al.donchian(df, 55) + raise ValueError(level) + + +def level_lookback_bars(level: str, bpd: int) -> int: + """Lookback per il conteggio equal-touch = finestra del livello stesso.""" + return {"prevday": 2 * bpd, "prevweek": 7 * bpd, "don20": 20, "don55": 55}[level] + + +# =========================================================================== +# EVENTI (sweep-and-reclaim) + outcome trade-level (overlap PERMESSO -> paired analysis) +# =========================================================================== +def _unfilled_fvg(h: np.ndarray, l: np.ndarray, i: int, d: int, price: float) -> bool: + """SHORT (d=-1): esiste FVG BULLISH (low[j] > high[j-2]) nelle ultime FVG_K barre con zona + (high[j-2], low[j]) interamente SOTTO il prezzo e mai riempita (nessuna barra dopo j e' + scesa fino al bordo inferiore). LONG (d=+1): simmetrico con FVG bearish sopra.""" + j0 = max(2, i - FVG_K) + for j in range(i - 1, j0 - 1, -1): + if d == -1 and l[j] > h[j - 2]: + zone_lo, zone_hi = h[j - 2], l[j] + if zone_hi < price and np.min(l[j + 1:i + 1]) > zone_lo: + return True + if d == 1 and h[j] < l[j - 2]: + zone_lo, zone_hi = h[j], l[j - 2] + if zone_lo > price and np.max(h[j + 1:i + 1]) < zone_hi: + return True + return False + + +def build_events(df: pd.DataFrame, level: str, shift_h: int = 0, + with_context: bool = True) -> pd.DataFrame: + """Tabella eventi sweep-and-reclaim con outcome trade-level (entry close[i], exit da i+1, + SL prioritario, fee 0.10% RT) + feature di contesto (eq/fvg/session). Overlap permesso: + ogni evento valutato indipendentemente -> confronto PAIRED filtro-vs-tutti pulito.""" + h = df["high"].values.astype(float) + l = df["low"].values.astype(float) + c = df["close"].values.astype(float) + n = len(c) + lvl_hi, lvl_lo = get_levels(df, level, shift_h) + a14 = al.atr(df, 14) + dt = pd.to_datetime(df["datetime"], utc=True) + hours = dt.dt.hour.values + bpd = al.bars_per_day(df) + lb = level_lookback_bars(level, bpd) + + sw_hi = np.isfinite(lvl_hi) & (h > lvl_hi) & (c < lvl_hi) + sw_lo = np.isfinite(lvl_lo) & (l < lvl_lo) & (c > lvl_lo) + both = sw_hi & sw_lo + sw_hi &= ~both + sw_lo &= ~both + + rows = [] + for i in np.where(sw_hi | sw_lo)[0]: + if i >= n - 1 or i < 60: + continue + d = -1 if sw_hi[i] else 1 + entry = c[i] + atr_i = a14[i] + if not np.isfinite(atr_i) or atr_i <= 0: + continue + if d == -1: + sl = h[i] + SL_ATR_BUF * atr_i + risk = sl - entry + tp = entry - R_MULT * risk + else: + sl = l[i] - SL_ATR_BUF * atr_i + risk = entry - sl + tp = entry + R_MULT * risk + if risk <= 0 or tp <= 0: + continue + jend = min(i + MAX_HOLD, n - 1) + exit_price = c[jend] + for j in range(i + 1, jend + 1): + if d == 1: + if l[j] <= sl: + exit_price = sl + break + if h[j] >= tp: + exit_price = tp + break + else: + if h[j] >= sl: + exit_price = sl + break + if l[j] <= tp: + exit_price = tp + break + exit_price = c[j] + gross = d * (exit_price - entry) / entry + L = lvl_hi[i] if d == -1 else lvl_lo[i] + row = dict(i=int(i), dir=int(d), entry=entry, sl=sl, tp=tp, level_px=float(L), + atr=float(atr_i), gross=gross, net=gross - FEE_RT) + if with_context: + j0 = max(0, i - lb) + tol = EQ_TOL_ATR * atr_i + touches = int(np.sum(np.abs((h if d == -1 else l)[j0:i] - L) <= tol)) + row["eq"] = touches >= EQ_MIN_TOUCH + row["fvg"] = _unfilled_fvg(h, l, int(i), int(d), float(entry)) + hr = int(hours[i]) + row["ses"] = next((s for s, (a, b) in SESSIONS.items() if a <= hr < b), "none") + rows.append(row) + ev = pd.DataFrame(rows) + if len(ev): + ev["dt"] = dt.values[ev["i"].values] + ev["hold"] = pd.to_datetime(ev["dt"], utc=True) >= HOLDOUT + ev["year"] = pd.to_datetime(ev["dt"], utc=True).dt.year + return ev + + +# =========================================================================== +# STRATEGIA (non-overlap, harness ufficiale) — metriche daily-step +# =========================================================================== +def entries_from(df: pd.DataFrame, sub: pd.DataFrame) -> list: + ent: list = [None] * len(df) + for row in sub.itertuples(): + ent[row.i] = dict(dir=int(row.dir), tp=float(row.tp), sl=float(row.sl), + max_bars=MAX_HOLD) + return ent + + +def strat_eval(df: pd.DataFrame, entries: list, fee_rt: float = FEE_RT) -> dict: + m = backtest_signals(df, entries, fee_rt=fee_rt, leverage=1.0) + idx = pd.DatetimeIndex(pd.to_datetime(df["datetime"], utc=True)) + eq = pd.Series(m.equity, index=idx) + d = eq.resample("1D").last().dropna().pct_change().dropna() + di, dh = d[d.index < HOLDOUT], d[d.index >= HOLDOUT] + return dict(n_trades=int(m.n_trades), wr=round(m.win_rate, 1), dd=round(m.max_dd, 4), + sh_full=round(al._sh(d), 3), sh_is=round(al._sh(di), 3), + sh_hold=round(al._sh(dh), 3), daily=d) + + +def apply_filter(ev: pd.DataFrame, filt: str | None) -> pd.DataFrame: + if not len(ev) or filt is None: + return ev + if filt == "eq": + return ev[ev["eq"]] + if filt == "fvg": + return ev[ev["fvg"]] + if filt.startswith("ses_"): + return ev[ev["ses"] == filt[4:]] + raise ValueError(filt) + + +def eval_trial(events: dict, tf: str, level: str, filt: str | None) -> dict: + per_asset, dailies = {}, {} + for a in ASSETS: + df = al.get(a, tf) + ev = events[(a, tf, level)] + sub = apply_filter(ev, filt) + r = strat_eval(df, entries_from(df, sub)) + yrs = (pd.to_datetime(df["datetime"].iloc[-1], utc=True) + - pd.to_datetime(df["datetime"].iloc[0], utc=True)).days / 365.25 + per_asset[a] = dict(r, n_ev=len(sub), ev_per_yr=round(len(sub) / yrs, 1), + exp_is=_exp(sub, False), exp_hold=_exp(sub, True)) + dailies[a] = r["daily"] + J = pd.concat(dailies, axis=1, join="inner").fillna(0.0) + comb = J.mean(axis=1) + ci, ch = comb[comb.index < HOLDOUT], comb[comb.index >= HOLDOUT] + return dict(tf=tf, level=level, filt=filt or "-", per_asset=per_asset, comb_daily=comb, + sh_is=round(al._sh(ci), 3), sh_hold=round(al._sh(ch), 3), + sh_full=round(al._sh(comb), 3), + min_sh_is=round(min(per_asset[a]["sh_is"] for a in ASSETS), 3), + min_sh_hold=round(min(per_asset[a]["sh_hold"] for a in ASSETS), 3)) + + +def _exp(sub: pd.DataFrame, hold: bool): + """Expectancy netta per trade (%) sullo slice IS/HOLD.""" + if not len(sub): + return None + s = sub[sub["hold"] == hold]["net"] + return round(float(s.mean()) * 100, 3) if len(s) else None + + +# =========================================================================== +# MAIN +# =========================================================================== +def main(): + print("=" * 100) + print("r0702 CRT CON CONTESTO — sweep-and-reclaim su livelli + filtri (EQ/FVG/sessione)") + print(f"setup fisso: entry close C2, SL estremo+/-{SL_ATR_BUF}*ATR14, TP R={R_MULT}:1, " + f"max_hold {MAX_HOLD} barre, fee {FEE_RT*100:.2f}% RT, SL prioritario same-bar") + print("=" * 100) + + # ---------- 0. CAUSALITY CHECK sui livelli (prefisso troncato) ---------- + print("\n[0] CAUSALITY CHECK livelli (ricalcolo su prefisso troncato, tail 200 barre)") + for level in LEVELS: + worst = 0.0 + for a in ASSETS: + df = al.get(a, "1h") + cut = int(len(df) * 0.9) + hi_f, lo_f = get_levels(df, level) + sub = df.iloc[:cut].reset_index(drop=True) + hi_s, lo_s = get_levels(sub, level) + for full, part in ((hi_f, hi_s), (lo_f, lo_s)): + x, y = np.nan_to_num(full[cut - 200:cut]), np.nan_to_num(part[cut - 200:cut]) + worst = max(worst, float(np.max(np.abs(x - y)))) + print(f" {level:<9s} max_tail_diff={worst:.10f} {'OK' if worst < 1e-9 else 'FAIL'}") + + # ---------- 1. EVENTI (cache) ---------- + events = {} + for a in ASSETS: + for tf in ("1h", "4h"): + df = al.get(a, tf) + for level in LEVELS: + events[(a, tf, level)] = build_events(df, level) + + # ---------- 2. TRIALS (22, definiti a priori) ---------- + trials = [] + for tf in ("1h", "4h"): + for level in LEVELS: + trials.append((tf, level, None)) + for tf in ("1h", "4h"): + for level in ("don20", "prevday"): + trials.append((tf, level, "eq")) + trials.append((tf, level, "fvg")) + for level in ("don20", "prevday"): + for ses in SESSIONS: + trials.append(("1h", level, f"ses_{ses}")) + assert len(trials) == 22 + + results = {} + for tf, level, filt in trials: + results[(tf, level, filt or "-")] = eval_trial(events, tf, level, filt) + + # ---------- 3. BASELINE INCONDIZIONATA (don20, nessun filtro) ---------- + print("\n[1] BASELINE INCONDIZIONATA — sweep-and-reclaim Donchian20, nessun contesto") + for tf in ("1h", "4h"): + r = results[(tf, "don20", "-")] + print(f" TF {tf}: comb Sharpe IS={r['sh_is']:+.2f} HOLD={r['sh_hold']:+.2f} " + f"FULL={r['sh_full']:+.2f}") + for a in ASSETS: + p = r["per_asset"][a] + print(f" {a}: ev/yr={p['ev_per_yr']:>6.1f} trades(no-overlap)={p['n_trades']:>5d} " + f"wr={p['wr']:>4.1f}% expIS={p['exp_is']}% expHOLD={p['exp_hold']}% " + f"Sh IS={p['sh_is']:+.2f} HOLD={p['sh_hold']:+.2f} DD={p['dd']*100:.1f}%") + + # ---------- 4. TUTTE LE CELLE (tabella) ---------- + print("\n[2] TUTTE LE 22 CELLE (comb 50/50, daily-step Sharpe; exp = %/trade netto)") + print(f" {'tf':<3s} {'level':<9s} {'filt':<9s} {'ShIS':>6s} {'ShHOLD':>7s} {'ShFULL':>7s} " + f"{'minShIS':>8s} {'minShHOLD':>9s} {'BTCexpIS':>9s} {'ETHexpIS':>9s} " + f"{'BTCexpH':>8s} {'ETHexpH':>8s} {'nBTC':>5s} {'nETH':>5s}") + for (tf, level, filt), r in sorted(results.items(), key=lambda kv: -kv[1]["sh_is"]): + pb, pe = r["per_asset"]["BTC"], r["per_asset"]["ETH"] + print(f" {tf:<3s} {level:<9s} {filt:<9s} {r['sh_is']:>+6.2f} {r['sh_hold']:>+7.2f} " + f"{r['sh_full']:>+7.2f} {r['min_sh_is']:>+8.2f} {r['min_sh_hold']:>+9.2f} " + f"{str(pb['exp_is']):>9s} {str(pe['exp_is']):>9s} " + f"{str(pb['exp_hold']):>8s} {str(pe['exp_hold']):>8s} " + f"{pb['n_ev']:>5d} {pe['n_ev']:>5d}") + + # ---------- 5. UPLIFT PAIRED dei filtri (stessi eventi, sottoinsieme vs tutti) ---------- + print("\n[3] UPLIFT PAIRED per filtro (expectancy %/trade: filtrato - tutti; stessi eventi)") + print(f" {'base':<16s} {'filtro':<9s} {'asset':<4s} {'slice':<5s} {'n_all':>6s} {'n_f':>5s} " + f"{'exp_all':>8s} {'exp_f':>8s} {'uplift':>8s}") + filt_names = ["eq", "fvg"] + [f"ses_{s}" for s in SESSIONS] + uplift_summary = {} + for tf in ("1h", "4h"): + for level in ("don20", "prevday"): + for filt in filt_names: + if filt.startswith("ses_") and tf != "1h": + continue + key = (tf, level, filt) + for a in ASSETS: + ev = events[(a, tf, level)] + sub = apply_filter(ev, filt) + for hold, lab in ((False, "IS"), (True, "HOLD")): + ea, ef = _exp(ev, hold), _exp(sub, hold) + na = int((ev["hold"] == hold).sum()) if len(ev) else 0 + nf = int((sub["hold"] == hold).sum()) if len(sub) else 0 + up = round(ef - ea, 3) if (ea is not None and ef is not None) else None + uplift_summary.setdefault(key, []).append((a, lab, up)) + print(f" {tf+'/'+level:<16s} {filt:<9s} {a:<4s} {lab:<5s} {na:>6d} " + f"{nf:>5d} {str(ea):>8s} {str(ef):>8s} {str(up):>8s}") + print("\n Consistenza uplift per filtro (positivo su TUTTE le 4 slice asset x IS/HOLD?):") + for key, ups in uplift_summary.items(): + vals = [u for (_, _, u) in ups if u is not None] + n_pos = sum(1 for u in vals if u > 0) + print(f" {key[0]}/{key[1]}+{key[2]:<9s}: {n_pos}/{len(vals)} slice positive " + f"{'<-- consistente' if vals and n_pos == len(vals) else ''}") + + # ---------- 6. SELEZIONE IN-SAMPLE + DSR ---------- + all_sr = [r["sh_full"] for r in results.values()] + chosen_key = max(results, key=lambda k: results[k]["sh_is"]) + ch = results[chosen_key] + dsr, sr0 = al.deflated_sharpe(ch["sh_full"], all_sr, ch["comb_daily"]) + print(f"\n[4] SELEZIONE IN-SAMPLE-ONLY (pre-2025) su {len(trials)} trial") + print(f" best-IS: {chosen_key} ShIS={ch['sh_is']:+.2f} ShHOLD={ch['sh_hold']:+.2f} " + f"ShFULL={ch['sh_full']:+.2f}") + print(f" deflated Sharpe (n_trials={len(all_sr)}): DSR={dsr:.3f} " + f"(PASS>=0.95) expected-null-max Sharpe={sr0:.2f}") + + # fee sweep sul best-IS e sulla baseline don20/1h + print("\n[5] FEE SWEEP (Sharpe FULL comb per fee RT)") + for key in {chosen_key, ("1h", "don20", "-")}: + tf, level, filt = key + row = [] + for fee in (0.0, 0.0005, 0.001, 0.0015, 0.002): + dailies = {} + for a in ASSETS: + df = al.get(a, tf) + sub = apply_filter(events[(a, tf, level)], None if filt == "-" else filt) + # ricalcola net eventi con fee diversa e' lineare; per la strategia rifacciamo il bt + dailies[a] = strat_eval(df, entries_from(df, sub), fee_rt=fee)["daily"] + J = pd.concat(dailies, axis=1, join="inner").fillna(0.0) + row.append(f"{fee*100:.2f}%RT:{al._sh(J.mean(axis=1)):+.2f}") + print(f" {key}: " + " ".join(row)) + + # ---------- 7. ANCHOR-SHIFT sulle celle sessione (+/-2/4h) ---------- + print("\n[6] ANCHOR-SHIFT celle sessione (label ora spostata; uplift expectancy IS per shift)") + for level in ("don20", "prevday"): + for ses in SESSIONS: + per_shift = {} + for sh in (-4, -2, 0, 2, 4): + ups = [] + for a in ASSETS: + ev = events[(a, "1h", level)] + if not len(ev): + continue + hrs = (pd.to_datetime(ev["dt"], utc=True).dt.hour + sh) % 24 + a_, b_ = SESSIONS[ses] + sub = ev[(hrs >= a_) & (hrs < b_)] + ea, ef = _exp(ev, False), _exp(sub, False) + if ea is not None and ef is not None: + ups.append(ef - ea) + per_shift[sh] = round(float(np.mean(ups)), 3) if ups else None + vals = [v for v in per_shift.values() if v is not None] + flip = vals and (max(vals) > 0 > min(vals)) and (max(vals) - min(vals)) > 0.05 + verd = "ARTIFACT-RISK(flip)" if flip else \ + ("stabile-pos" if vals and min(vals) > 0 else + "stabile-neg/nullo" if vals and max(vals) <= 0 else "misto-debole") + print(f" {level}+{ses:<5s}: " + + " ".join(f"{k:+d}h:{v}" for k, v in per_shift.items()) + f" -> {verd}") + + # ---------- 8. DAY-BOUNDARY SHIFT sul fade prevday base (1h) ---------- + print("\n[7] DAY-BOUNDARY SHIFT su fade prevday base 1h (livelli ricostruiti col giorno spostato)") + for sh in (0, 2, 4, 8, 12): + dailies = {} + for a in ASSETS: + df = al.get(a, "1h") + ev = build_events(df, "prevday", shift_h=sh, with_context=False) + dailies[a] = strat_eval(df, entries_from(df, ev))["daily"] + J = pd.concat(dailies, axis=1, join="inner").fillna(0.0) + comb = J.mean(axis=1) + ci = comb[comb.index < HOLDOUT] + print(f" shift +{sh:>2d}h: Sh IS={al._sh(ci):+.2f} FULL={al._sh(comb):+.2f}") + + # ---------- 9. FADE vs FOLLOW sui livelli prior-day (lead esistente) ---------- + print("\n[8] FADE (questo filone, prevday base 1h) vs FOLLOW (lead prevday_breakout congelato)") + fol = {} + for a in ASSETS: + df = al.get(a, "1h") + evw = al.eval_weights(df, prevday_follow_target(df)) + fol[a] = pd.Series(evw["net"], index=evw["idx"]) + Jf = pd.concat(fol, axis=1, join="inner").fillna(0.0) + follow_d = al._to_daily(0.5 * Jf["BTC"] + 0.5 * Jf["ETH"]) + fade_d = results[("1h", "prevday", "-")]["comb_daily"] + JJ = pd.concat({"fade": fade_d, "follow": follow_d}, axis=1, join="inner").dropna() + JH = JJ[JJ.index >= HOLDOUT] + JI = JJ[JJ.index < HOLDOUT] + print(f" corr daily fade-follow: FULL={JJ['fade'].corr(JJ['follow']):+.3f} " + f"HOLD={JH['fade'].corr(JH['follow']):+.3f}") + print(f" Sharpe IS : fade={al._sh(JI['fade']):+.2f} follow={al._sh(JI['follow']):+.2f}") + print(f" Sharpe HOLD: fade={al._sh(JH['fade']):+.2f} follow={al._sh(JH['follow']):+.2f}") + print(" per anno (Sharpe fade | follow):") + for y in sorted(set(JJ.index.year)): + sub = JJ[JJ.index.year == y] + if len(sub) > 40: + print(f" {y}: {al._sh(sub['fade']):+.2f} | {al._sh(sub['follow']):+.2f}") + + # ---------- 10. MARGINAL vs TP01 (solo se il best-IS regge) ---------- + if ch["sh_full"] >= 0.5 and ch["sh_is"] >= 0.5: + print("\n[9] MARGINAL vs TP01 (best-IS regge >=0.5 -> gate)") + m = al.marginal_vs_tp01(ch["comb_daily"]) + print(f" verdict={m.get('marginal_verdict')} corr_full={m.get('corr_full')} " + f"uplift w25 full={m['blends']['w25']['uplift_full']:+.3f} " + f"hold={m['blends']['w25']['uplift_hold']}") + print(f" has_insample_edge={m.get('has_insample_edge')} is_hedge={m.get('is_hedge')} " + f"robust_oos={m.get('robust_oos')} multicut={m.get('multicut_uplift')}") + else: + print(f"\n[9] MARGINAL vs TP01: SALTATO — best-IS Sharpe FULL={ch['sh_full']:+.2f} / " + f"IS={ch['sh_is']:+.2f} sotto la soglia 0.5 standalone") + + print("\nFine. Nessun file scritto fuori da questo script; selezione solo in-sample.") + + +if __name__ == "__main__": + main() diff --git a/scripts/research/r0702_crt_mtf.py b/scripts/research/r0702_crt_mtf.py new file mode 100644 index 0000000..7380d58 --- /dev/null +++ b/scripts/research/r0702_crt_mtf.py @@ -0,0 +1,579 @@ +#!/usr/bin/env python +"""r0702_crt_mtf.py — CRT (Candle Range Theory) MULTI-TIMEFRAME — filone 2026-07-02. + +TESI DA TESTARE (scuola CRT): il pattern a 3 candele + C1 = candela-range forte; C2 = sweep di UN estremo di C1 con close back-inside (presa di + liquidita'); C3 = ingresso CONTRO il breakout +renderebbe molto di piu' eseguito MULTI-TF: struttura su TF alto (4h/1h), ingresso sul TF basso +(15m/5m) al RITEST della zona violata -> stop dietro lo swing del TF basso (piu' stretto del +"dietro l'estremo di C2" single-TF) -> R:R da ~1.3 a ~3+. + +DISEGNO SPERIMENTALE: confronto CONTROLLATO/PAIRED sugli STESSI pattern C1-C2, tre esecuzioni: + (i) BASE single-TF: entry a open di C3, stop dietro l'estremo di C2, target estremo opposto C1 + (ii) MTF ritest della zona + trigger di conferma sul TF basso, stop dietro lo swing basso + (iii) NOTRIG ritest puro (entry al primo tocco della zona), senza conferma bassa +Tutte e tre simulate sulla STESSA griglia di barre del TF basso (fill intrabar identici, +conservativi: SL prima di TP nella stessa barra bassa). Fee 0.10% RT + sweep 0/0.10/0.20. + +DEFINIZIONI FISSATE A PRIORI (dichiarate prima di guardare i risultati, nessuna sensibilita' qui; +la sensibilita' della detection e' del filone base single-TF): + - C1 forte: range >= 1.2 * ATR14 del TF alto (UNA definizione; body/range NON usato). + - C2: rompe UN SOLO estremo di C1 (doppio sweep = skip) e chiude DENTRO il range di C1. + - Finestra: 1 barra del TF alto dopo la chiusura di C2 (la "C3"). + - Max hold: 20 barre high-TF dall'apertura della finestra, poi exit a market al close + (identico per tutte le varianti -> confronto pulito). + - Invalidation (solo MTF/NOTRIG): se PRIMA del trigger il prezzo supera l'estremo di C2 + (>=, conservativo), setup invalidato -> no trade (la BASE nello stesso caso + viene semplicemente stoppata: e' la differenza strutturale fra le esecuzioni). + - R:R >= 1.3 all'entry per MTF/NOTRIG (parte della tesi CRT-MTF). La BASE non e' filtrata + (e' l'esecuzione classica single-TF). + - Sizing: 1.0x nozionale per trade; book SEQUENZIALE per asset (1 trade aperto alla volta) + per la serie daily (Sharpe/DD); expectancy per-trade su TUTTI i pattern (indip.). + +GRIGLIA (unica, chiusa a priori): d in {0.10, 0.25} x trigger in {closeback, sweeprec} per MTF; +d in {0.10, 0.25} per NOTRIG. Selezione cella SOLO in-sample (<2025-01-01). Trials per DSR = +(1 base + 4 MTF + 2 NOTRIG) x 2 coppie TF = 14. + +Trigger meccanici sul TF basso (per short, simmetrico per long); L = estremo C1 violato, +zona = [L - d*ATR14_alto, L]: + - closeback: dopo che una barra bassa ha TOCCATO la zona (high >= L - d*ATR), la prima barra + bassa che CHIUDE sotto L -> entry al suo close. + - sweeprec: barra bassa j che tocca la zona E sweep del massimo della barra bassa precedente + (high[j] > high[j-1]) E chiude sotto high[j-1] E sotto L -> entry al close di j. +Stop MTF/NOTRIG = estremo dello swing basso (max high dall'apertura della finestra alla barra +del trigger inclusa). Target (tutte): estremo opposto di C1. + +Esecuzione: uv run python scripts/research/r0702_crt_mtf.py +NON tocca src/, config/, scripts/live/. Nessun file scritto. +""" +from __future__ import annotations + +import sys +import time +from collections import Counter + +sys.path.insert(0, "/opt/docker/PythagorasGoal/scripts/research/alt") + +import numpy as np +import pandas as pd + +import altlib as al + +# ---------------------------------------------------------------- config (a priori) +PAIRS = (("4h", "15m"), ("1h", "5m")) +ASSETS = ("BTC", "ETH") +D_GRID = (0.10, 0.25) +TRIGGERS = ("closeback", "sweeprec") +MAXHOLD_HTF = 20 # barre high-TF di holding max, dalla apertura della finestra C3 +RR_MIN = 1.3 # filtro R:R all'entry (solo MTF/NOTRIG) +ATR_MULT = 1.2 # C1 forte: range >= 1.2*ATR14 (definizione unica) +FEE_RT = 0.001 # 0.10% round-trip +FEE_SWEEP_RT = (0.0, 0.001, 0.002) +TF_MS = {"5m": 300_000, "15m": 900_000, "1h": 3_600_000, "4h": 14_400_000} +HOLDOUT = al.HOLDOUT +HOLDOUT_MS = int(HOLDOUT.value // 10**6) +CAPITAL = 600.0 +LEV_CAP = 2.0 +MIN_ORDER = 5.0 + + +# ---------------------------------------------------------------- detection (vettoriale) +def detect_patterns(dfh: pd.DataFrame, tf_hi: str) -> list[dict]: + """CRT C1-C2 sul TF alto. Pattern noto alla CHIUSURA di C2 (causale: usa solo barre <= C2).""" + ts = dfh["timestamp"].astype("int64").values + o = dfh["open"].values.astype(float) + h = dfh["high"].values.astype(float) + l = dfh["low"].values.astype(float) + c = dfh["close"].values.astype(float) + a = al.atr(dfh, 14) + rng = h - l + strong = rng >= ATR_MULT * a + + h1 = np.roll(h, 1); l1 = np.roll(l, 1); s1 = np.roll(strong, 1) + up = s1 & (h > h1) & ~(l < l1) & (c <= h1) & (c >= l1) # sweep del massimo di C1 -> SHORT + dn = s1 & (l < l1) & ~(h > h1) & (c >= l1) & (c <= h1) # sweep del minimo di C1 -> LONG + idx = np.where(up | dn)[0] + tf_ms = TF_MS[tf_hi] + pats = [] + for i in idx: + if i < 20: # warm-up ATR + continue + if up[i]: + d, level, target, c2ext = -1, h1[i], l1[i], h[i] + else: + d, level, target, c2ext = +1, l1[i], h1[i], l[i] + pats.append(dict(i=int(i), dir=d, level=float(level), target=float(target), + c2ext=float(c2ext), atr=float(a[i]), + win_open=int(ts[i] + tf_ms), + win_close=int(ts[i] + 2 * tf_ms), + hold_end=int(ts[i] + (1 + MAXHOLD_HTF) * tf_ms))) + return pats + + +# ---------------------------------------------------------------- low-TF arrays +class Low: + __slots__ = ("ts", "o", "h", "l", "c", "tsclose", "n") + + def __init__(self, df: pd.DataFrame, tf_lo: str): + self.ts = df["timestamp"].astype("int64").values + self.o = df["open"].values.astype(float) + self.h = df["high"].values.astype(float) + self.l = df["low"].values.astype(float) + self.c = df["close"].values.astype(float) + self.tsclose = self.ts + TF_MS[tf_lo] + self.n = len(self.ts) + + +def scan_exit(L: Low, j0: int, j1: int, dr: int, entry_ts: int, + stop: float, target: float): + """Barre j0..j1-1; conservativo: SL prima di TP nella stessa barra. Ritorna (px, ts, kind).""" + Lh, Ll, Lc, Ltsc = L.h, L.l, L.c, L.tsclose + for j in range(j0, j1): + if dr < 0: + if Lh[j] >= stop: + return stop, int(Ltsc[j]), "SL" + if Ll[j] <= target: + return target, int(Ltsc[j]), "TP" + else: + if Ll[j] <= stop: + return stop, int(Ltsc[j]), "SL" + if Lh[j] >= target: + return target, int(Ltsc[j]), "TP" + if j1 - 1 < j0: + return None, entry_ts, "NOBARS" + return float(Lc[j1 - 1]), int(Ltsc[j1 - 1]), "TIME" + + +def _mk_trade(p, entry, entry_ts, stop, exitp, exit_ts, kind, jt=None, j1=None): + dr = p["dir"] + risk = abs(stop - entry) / entry + gross = dr * (exitp / entry - 1.0) + rr = (abs(entry - p["target"]) / abs(stop - entry)) if stop != entry else np.inf + return dict(ok=True, dir=dr, entry=entry, stop=stop, target=p["target"], + risk=risk, rr=rr, gross=gross, entry_ts=int(entry_ts), + exit_ts=int(exit_ts), kind=kind, jt=jt, j1=j1) + + +def trade_base(p: dict, L: Low): + """(i) BASE: entry a open C3 (= prima barra bassa della finestra), stop dietro estremo C2.""" + j0 = int(np.searchsorted(L.ts, p["win_open"])) + if j0 >= L.n or L.ts[j0] >= p["win_close"]: + return dict(ok=False, reason="nodata") + entry = float(L.o[j0]) + dr, stop, target = p["dir"], p["c2ext"], p["target"] + if (dr < 0 and not (target < entry < stop)) or (dr > 0 and not (stop < entry < target)): + return dict(ok=False, reason="degenerate") + j1 = int(np.searchsorted(L.ts, p["hold_end"])) + exitp, exit_ts, kind = scan_exit(L, j0, j1, dr, int(L.ts[j0]), stop, target) + if exitp is None: + return dict(ok=False, reason="nodata") + return _mk_trade(p, entry, L.ts[j0], stop, exitp, exit_ts, kind, jt=j0, j1=j1) + + +def trade_mtf(p: dict, L: Low, d_mult: float, trigger: str | None): + """(ii) MTF con trigger / (iii) NOTRIG (trigger=None): ritest della zona nella finestra C3.""" + j0 = int(np.searchsorted(L.ts, p["win_open"])) + if j0 >= L.n or L.ts[j0] >= p["win_close"]: + return dict(ok=False, reason="nodata") + jw = int(np.searchsorted(L.ts, p["win_close"])) + dr, level, c2ext, target = p["dir"], p["level"], p["c2ext"], p["target"] + zone = d_mult * p["atr"] + Lh, Ll, Lc = L.h, L.l, L.c + touched = False + jt = -1 + if dr < 0: + swing = -np.inf + for j in range(j0, jw): + if Lh[j] > swing: + swing = Lh[j] + if Lh[j] >= c2ext: # struttura violata prima del trigger + return dict(ok=False, reason="invalidated") + if Lh[j] >= level - zone: + touched = True + if touched: + if trigger is None: + jt = j; break + if trigger == "closeback" and Lc[j] < level: + jt = j; break + if (trigger == "sweeprec" and j >= 1 and Lh[j] >= level - zone + and Lh[j] > Lh[j - 1] and Lc[j] < Lh[j - 1] and Lc[j] < level): + jt = j; break + if jt < 0: + return dict(ok=False, reason=("notrigger" if touched else "noretest")) + stop = float(max(swing, Lh[jt])) + entry = float(Lc[jt]) + if not (target < entry < stop): + return dict(ok=False, reason="degenerate") + else: + swing = np.inf + for j in range(j0, jw): + if Ll[j] < swing: + swing = Ll[j] + if Ll[j] <= c2ext: + return dict(ok=False, reason="invalidated") + if Ll[j] <= level + zone: + touched = True + if touched: + if trigger is None: + jt = j; break + if trigger == "closeback" and Lc[j] > level: + jt = j; break + if (trigger == "sweeprec" and j >= 1 and Ll[j] <= level + zone + and Ll[j] < Ll[j - 1] and Lc[j] > Ll[j - 1] and Lc[j] > level): + jt = j; break + if jt < 0: + return dict(ok=False, reason=("notrigger" if touched else "noretest")) + stop = float(min(swing, Ll[jt])) + entry = float(Lc[jt]) + if not (stop < entry < target): + return dict(ok=False, reason="degenerate") + rr = abs(entry - target) / abs(stop - entry) + if rr < RR_MIN: + return dict(ok=False, reason="rrfail") + j1 = int(np.searchsorted(L.ts, p["hold_end"])) + exitp, exit_ts, kind = scan_exit(L, jt + 1, j1, dr, int(L.tsclose[jt]), stop, target) + if exitp is None: # trigger sull'ultima barra: flat, -fee + exitp, exit_ts, kind = entry, int(L.tsclose[jt]), "NOBARS" + return _mk_trade(p, entry, L.tsclose[jt], stop, exitp, exit_ts, kind, jt=jt, j1=j1) + + +# ---------------------------------------------------------------- stats & book +def trade_stats(trades: list[dict], fee_rt: float = FEE_RT) -> dict: + tr = [t for t in trades if t and t.get("ok")] + if not tr: + return dict(n=0, exp_bps=np.nan, wr=np.nan, avgR=np.nan, med_risk=np.nan, avg_rr=np.nan) + nets = np.array([t["gross"] - fee_rt for t in tr]) + Rs = np.array([(t["gross"] - fee_rt) / t["risk"] for t in tr if t["risk"] > 0]) + return dict(n=len(tr), exp_bps=float(nets.mean() * 1e4), wr=float((nets > 0).mean() * 100), + avgR=float(Rs.mean()) if len(Rs) else np.nan, + med_risk=float(np.median([t["risk"] for t in tr]) * 100), + avg_rr=float(np.mean([min(t["rr"], 50.0) for t in tr]))) + + +def seq_filter(trades: list[dict]) -> list[dict]: + out, last = [], -1 + for t in sorted((t for t in trades if t and t.get("ok")), key=lambda x: x["entry_ts"]): + if t["entry_ts"] >= last: + out.append(t) + last = t["exit_ts"] + return out + + +def daily_series(seq_trades: list[dict], span: tuple[int, int], fee_rt: float = FEE_RT) -> pd.Series: + idx = pd.date_range(pd.Timestamp(span[0], unit="ms", tz="UTC").normalize(), + pd.Timestamp(span[1], unit="ms", tz="UTC").normalize(), freq="D") + s = pd.Series(0.0, index=idx) + for t in seq_trades: + d = pd.Timestamp(t["exit_ts"], unit="ms", tz="UTC").normalize() + if d in s.index: + s[d] += t["gross"] - fee_rt + return s + + +def sh_dd(s: pd.Series) -> tuple[float, float]: + sharpe = al._sh(s) + eq = np.cumprod(1.0 + s.values) + pk = np.maximum.accumulate(eq) + dd = float(np.max((pk - eq) / pk)) if len(eq) else 0.0 + return sharpe, dd + + +def portfolio_daily(res_pair: dict, key, spans: dict, fee_rt: float = FEE_RT) -> pd.Series: + per = [] + for a in ASSETS: + seq = seq_filter(res_pair[a][key]) + per.append(daily_series(seq, spans[a], fee_rt)) + J = pd.concat(per, axis=1).fillna(0.0) + return 0.5 * J.iloc[:, 0] + 0.5 * J.iloc[:, 1] + + +def split_hold(trades: list[dict]) -> tuple[list, list]: + ins = [t for t in trades if t and t.get("ok") and t["entry_ts"] < HOLDOUT_MS] + hold = [t for t in trades if t and t.get("ok") and t["entry_ts"] >= HOLDOUT_MS] + return ins, hold + + +# ---------------------------------------------------------------- delayed execution (cron orario) +def delayed_eval(trades: list[dict], L: Low, fee_rt: float = FEE_RT) -> dict: + """Il book live gira ogni ora: il segnale (close barra bassa) viene eseguito alla PRIMA + chiusura di barra bassa sulla griglia oraria successiva. Se nel frattempo SL/TP e' gia' + stato attraversato -> nessun ingresso (skip). Ritorna expectancy originale vs ritardata.""" + orig, dela, delays = [], [], [] + n_skip_sl = n_skip_tp = n_missed_window = 0 + for t in trades: + if not (t and t.get("ok")): + continue + ts_e = t["entry_ts"] + boundary = ((ts_e + 3_599_999) // 3_600_000) * 3_600_000 + delays.append((boundary - ts_e) / 60_000.0) + if boundary == ts_e: + orig.append(t["gross"] - fee_rt) + dela.append(t["gross"] - fee_rt) + continue + jb = int(np.searchsorted(L.tsclose, boundary)) + j1 = t["j1"] + if jb >= L.n or jb >= j1: + n_missed_window += 1 + orig.append(t["gross"] - fee_rt) + continue + dr, stop, target = t["dir"], t["stop"], t["target"] + crossed = None + for j in range(t["jt"] + 1, jb + 1): + if dr < 0: + if L.h[j] >= stop: + crossed = "SL"; break + if L.l[j] <= target: + crossed = "TP"; break + else: + if L.l[j] <= stop: + crossed = "SL"; break + if L.h[j] >= target: + crossed = "TP"; break + orig.append(t["gross"] - fee_rt) + if crossed == "SL": + n_skip_sl += 1 + continue + if crossed == "TP": + n_skip_tp += 1 + continue + entry2 = float(L.c[jb]) + if (dr < 0 and not (target < entry2 < stop)) or (dr > 0 and not (stop < entry2 < target)): + n_skip_sl += 1 + continue + exitp, _, _ = scan_exit(L, jb + 1, j1, dr, int(L.tsclose[jb]), stop, target) + if exitp is None: + exitp = entry2 + dela.append(dr * (exitp / entry2 - 1.0) - fee_rt) + n_sig = len(orig) + return dict(n_signals=n_sig, mean_delay_min=float(np.mean(delays)) if delays else np.nan, + n_entered=len(dela), n_skip_sl=n_skip_sl, n_skip_tp=n_skip_tp, + n_missed_window=n_missed_window, + exp_orig_bps=float(np.mean(orig) * 1e4) if orig else np.nan, + exp_delayed_entered_bps=float(np.mean(dela) * 1e4) if dela else np.nan, + exp_delayed_per_signal_bps=float(np.sum(dela) / n_sig * 1e4) if n_sig else np.nan) + + +# ---------------------------------------------------------------- main +def key_label(key) -> str: + if key == ("base",): + return "BASE single-TF " + if key[0] == "mtf": + return f"MTF d={key[1]:.2f} {key[2]:<9s}" + return f"NOTRIG d={key[1]:.2f} " + + +def main(): + t0 = time.time() + print("=" * 100) + print("r0702 CRT MULTI-TIMEFRAME — struttura su TF alto, ingresso su TF basso (paired vs base)") + print(f"C1 forte: range>={ATR_MULT}*ATR14 | maxhold {MAXHOLD_HTF} barre HTF | RR>={RR_MIN} (MTF) " + f"| fee {FEE_RT*1e4:.0f}bps RT | hold-out >= {HOLDOUT.date()}") + print("=" * 100) + + all_trial_sharpes = [] # per DSR: full Sharpe di OGNI (pair, variant-cell) + chosen_summaries = [] # per selezione finale cross-pair + + for tf_hi, tf_lo in PAIRS: + print(f"\n{'#'*100}\n### COPPIA {tf_hi} -> {tf_lo}\n{'#'*100}") + res: dict[str, dict] = {} + spans: dict[str, tuple[int, int]] = {} + reasons: dict[str, dict] = {} + n_pats: dict[str, int] = {} + + variant_keys = [("base",)] + [("mtf", d, tr) for d in D_GRID for tr in TRIGGERS] \ + + [("notrig", d) for d in D_GRID] + + for a in ASSETS: + dfh = al.get(a, tf_hi) + L = Low(al.get(a, tf_lo), tf_lo) + spans[a] = (int(L.ts[0]), int(L.tsclose[-1])) + pats = detect_patterns(dfh, tf_hi) + n_pats[a] = len(pats) + res[a] = {} + reasons[a] = {} + for key in variant_keys: + outs = [] + for p in pats: + if key[0] == "base": + outs.append(trade_base(p, L)) + elif key[0] == "mtf": + outs.append(trade_mtf(p, L, key[1], key[2])) + else: + outs.append(trade_mtf(p, L, key[1], None)) + res[a][key] = outs + reasons[a][key] = Counter(t.get("reason") for t in outs if not t.get("ok")) + print(f" {a}: {len(pats)} pattern C1-C2 su {tf_hi} " + f"(short={sum(1 for p in pats if p['dir'] < 0)}, long={sum(1 for p in pats if p['dir'] > 0)})") + + # ------- tabella varianti: per-trade (tutti i pattern, indip.) + book sequenziale 50/50 + print(f"\n --- VARIANTI (pooled BTC+ETH; per-trade su tutti i pattern; Sharpe/DD su book " + f"sequenziale 50/50, daily) ---") + hdr = (f" {'variante':<24s} | {'n_FULL':>6s} {'exp(bps)':>8s} {'WR%':>5s} {'avgR':>6s} " + f"{'RRm':>5s} {'Sh_F':>6s} {'DD_F%':>6s} | {'n_H':>5s} {'expH':>8s} {'WRH':>5s} " + f"{'Sh_H':>6s} | {'riskMed%':>8s}") + print(hdr) + table = {} + for key in variant_keys: + pooled = res["BTC"][key] + res["ETH"][key] + ins, hold = split_hold(pooled) + st_f = trade_stats(ins + hold) + st_h = trade_stats(hold) + port = portfolio_daily(res, key, spans) + sh_f, dd_f = sh_dd(port) + ph = port[port.index >= HOLDOUT] + sh_h, _ = sh_dd(ph) if len(ph) > 30 else (np.nan, np.nan) + pi = port[port.index < HOLDOUT] + sh_is, _ = sh_dd(pi) if len(pi) > 30 else (np.nan, np.nan) + st_is = trade_stats(ins) + table[key] = dict(st_f=st_f, st_h=st_h, st_is=st_is, sh_f=sh_f, dd_f=dd_f, + sh_h=sh_h, sh_is=sh_is, port=port) + all_trial_sharpes.append(sh_f) + print(f" {key_label(key)} | {st_f['n']:>6d} {st_f['exp_bps']:>8.1f} {st_f['wr']:>5.1f} " + f"{st_f['avgR']:>6.2f} {st_f['avg_rr']:>5.1f} {sh_f:>6.2f} {dd_f*100:>6.1f} | " + f"{st_h['n']:>5d} {st_h['exp_bps']:>8.1f} {st_h['wr']:>5.1f} {sh_h:>6.2f} | " + f"{st_f['med_risk']:>8.3f}") + + # ------- quota pattern senza ritest / invalidati / rr-fail (per cella MTF) + print("\n --- FUNNEL pattern -> trade (pooled, % dei pattern) ---") + for key in variant_keys[1:]: + cnt = reasons["BTC"][key] + reasons["ETH"][key] + tot = n_pats["BTC"] + n_pats["ETH"] + n_tr = table[key]["st_f"]["n"] + print(f" {key_label(key)} | trade {n_tr:>5d} ({n_tr/tot*100:4.1f}%) | " + f"no-ritest {cnt.get('noretest', 0)/tot*100:4.1f}% | " + f"no-trigger {cnt.get('notrigger', 0)/tot*100:4.1f}% | " + f"invalidato {cnt.get('invalidated', 0)/tot*100:4.1f}% | " + f"RR<{RR_MIN} {cnt.get('rrfail', 0)/tot*100:4.1f}% | " + f"altro {sum(v for k, v in cnt.items() if k in ('nodata', 'degenerate'))/tot*100:4.1f}%") + + # ------- selezione cella SOLO in-sample (<2025) + mtf_keys = [k for k in variant_keys if k[0] == "mtf"] + ntg_keys = [k for k in variant_keys if k[0] == "notrig"] + + def is_score(k): + v = table[k]["sh_is"] + return v if np.isfinite(v) else -9 + + best_mtf = max(mtf_keys, key=is_score) + best_ntg = max(ntg_keys, key=is_score) + print(f"\n --- SELEZIONE IN-SAMPLE (<2025, Sharpe book 50/50) ---") + for k in mtf_keys + ntg_keys: + mark = " <== scelta" if k in (best_mtf, best_ntg) else "" + print(f" {key_label(k)} | Sh_IS={table[k]['sh_is']:>6.2f} exp_IS={table[k]['st_is']['exp_bps']:>7.1f}bps " + f"(n_IS={table[k]['st_is']['n']}){mark}") + print(f" BASE | Sh_IS={table[('base',)]['sh_is']:>6.2f} " + f"exp_IS={table[('base',)]['st_is']['exp_bps']:>7.1f}bps (n_IS={table[('base',)]['st_is']['n']})") + + # ------- confronto PAIRED sugli stessi pattern (subset dove TUTTE e 3 hanno tradato) + print(f"\n --- PAIRED sugli stessi pattern (BASE vs MTF{best_mtf[1:]} vs NOTRIG d={best_ntg[1]}) ---") + for label, mask_hold in (("FULL", None), ("HOLD", True)): + diffs_mb, diffs_nb = [], [] + rows = {k: [] for k in (("base",), best_mtf, best_ntg)} + for a in ASSETS: + for tb, tm, tn in zip(res[a][("base",)], res[a][best_mtf], res[a][best_ntg]): + if not (tb.get("ok") and tm.get("ok") and tn.get("ok")): + continue + if mask_hold and tm["entry_ts"] < HOLDOUT_MS: + continue + if mask_hold is None and False: + continue + rows[("base",)].append(tb) + rows[best_mtf].append(tm) + rows[best_ntg].append(tn) + diffs_mb.append((tm["gross"] - FEE_RT) - (tb["gross"] - FEE_RT)) + diffs_nb.append((tn["gross"] - FEE_RT) - (tb["gross"] - FEE_RT)) + n = len(diffs_mb) + if n < 5: + print(f" [{label}] n={n} — potenza statistica insufficiente per il paired") + continue + d_mb = np.array(diffs_mb); d_nb = np.array(diffs_nb) + t_mb = d_mb.mean() / (d_mb.std(ddof=1) / np.sqrt(n)) if d_mb.std() > 0 else np.nan + t_nb = d_nb.mean() / (d_nb.std(ddof=1) / np.sqrt(n)) if d_nb.std() > 0 else np.nan + print(f" [{label}] n_paired={n}") + for k in (("base",), best_mtf, best_ntg): + st = trade_stats(rows[k]) + print(f" {key_label(k)} | exp={st['exp_bps']:>7.1f}bps WR={st['wr']:>5.1f}% " + f"avgR={st['avgR']:>6.2f} RRmedio={st['avg_rr']:>4.1f} riskMed={st['med_risk']:.3f}%") + print(f" Δ(MTF-BASE) = {d_mb.mean()*1e4:>+7.1f}bps/trade t={t_mb:+.2f}") + print(f" Δ(NOTRIG-BASE)= {d_nb.mean()*1e4:>+7.1f}bps/trade t={t_nb:+.2f}") + + # ------- fee sweep (celle scelte + base) + print(f"\n --- FEE SWEEP (exp bps/trade FULL | Sharpe book) ---") + for k in (("base",), best_mtf, best_ntg): + parts = [] + for f in FEE_SWEEP_RT: + pooled = [t for t in res["BTC"][k] + res["ETH"][k] if t.get("ok")] + e = np.mean([t["gross"] - f for t in pooled]) * 1e4 if pooled else np.nan + shf, _ = sh_dd(portfolio_daily(res, k, spans, fee_rt=f)) + parts.append(f"{f*1e4:3.0f}bps: {e:+7.1f}bps/Sh {shf:+5.2f}") + print(f" {key_label(k)} | " + " | ".join(parts)) + + # ------- esecuzione ritardata alla griglia oraria (celle MTF scelte) + print(f"\n --- ESECUZIONE RITARDATA (cron orario) ---") + for k in (best_mtf, best_ntg): + agg = dict(n_signals=0, n_entered=0, n_skip_sl=0, n_skip_tp=0, n_missed_window=0) + wsum_o = wsum_d = wsum_ps = 0.0 + dsum = 0.0 + for a in ASSETS: + L = Low(al.get(a, tf_lo), tf_lo) + d = delayed_eval(res[a][k], L) + for kk in agg: + agg[kk] += d[kk] + wsum_o += d["exp_orig_bps"] * d["n_signals"] if d["n_signals"] else 0 + wsum_d += d["exp_delayed_entered_bps"] * d["n_entered"] if d["n_entered"] else 0 + wsum_ps += d["exp_delayed_per_signal_bps"] * d["n_signals"] if d["n_signals"] else 0 + dsum += d["mean_delay_min"] * d["n_signals"] if d["n_signals"] else 0 + ns, ne = agg["n_signals"], agg["n_entered"] + print(f" {key_label(k)} | segnali {ns} | gap medio {dsum/ns if ns else np.nan:.1f}min | " + f"entrati {ne} ({ne/ns*100 if ns else 0:.0f}%) skipSL {agg['n_skip_sl']} " + f"skipTP {agg['n_skip_tp']} persi-finestra {agg['n_missed_window']}") + print(f" exp originale {wsum_o/ns if ns else np.nan:+.1f}bps/trade -> ritardata " + f"{wsum_d/ne if ne else np.nan:+.1f}bps/trade (entrati) | per-SEGNALE " + f"{wsum_ps/ns if ns else np.nan:+.1f}bps") + + # ------- executability a $600 + print(f"\n --- EXECUTABILITY $600 (cap leva {LEV_CAP}x, min order ${MIN_ORDER}) ---") + for k in (("base",), best_mtf, best_ntg): + tr = [t for t in res["BTC"][k] + res["ETH"][k] if t.get("ok")] + if not tr: + continue + risks = np.array([t["risk"] for t in tr]) * 100 + med = float(np.median(risks)) + lev_1pct = 1.0 / med if med > 0 else np.inf + yrs = (spans["BTC"][1] - spans["BTC"][0]) / (365.25 * 86400e3) + tpy = len(seq_filter(res["BTC"][k])) / yrs + len(seq_filter(res["ETH"][k])) / yrs + print(f" {key_label(k)} | stopMed {med:.3f}% (p25 {np.percentile(risks, 25):.3f} / " + f"p75 {np.percentile(risks, 75):.3f}) | leva per rischio-1% = {lev_1pct:.1f}x " + f"-> CAP {LEV_CAP}x: rischio/trade {LEV_CAP*med:.2f}% (${CAPITAL*LEV_CAP*med/100:.1f}) " + f"| nozionale ${CAPITAL*LEV_CAP:.0f} > min ${MIN_ORDER} OK | ~{tpy:.0f} trade/anno (seq)") + + chosen_summaries.append(dict(pair=f"{tf_hi}->{tf_lo}", key=best_mtf, table=table, + res=res, spans=spans, tf_lo=tf_lo)) + + # ---------------- DSR sul candidato scelto in-sample fra TUTTI i trial + print(f"\n{'='*100}\n### GATE STATISTICI GLOBALI\n{'='*100}") + best = max(chosen_summaries, key=lambda cs: cs["table"][cs["key"]]["sh_is"] + if np.isfinite(cs["table"][cs["key"]]["sh_is"]) else -9) + bt = best["table"][best["key"]] + print(f"Candidato scelto (best in-sample fra le celle MTF): {best['pair']} {key_label(best['key'])} " + f"| Sh_IS={bt['sh_is']:.2f} Sh_FULL={bt['sh_f']:.2f} Sh_HOLD={bt['sh_h']:.2f}") + valid_trials = [s for s in all_trial_sharpes if np.isfinite(s)] + dsr, sr0 = al.deflated_sharpe(bt["sh_f"], valid_trials, bt["port"].values) + print(f"Deflated Sharpe (n_trials={len(valid_trials)}): DSR={dsr:.3f} " + f"(expected null max Sharpe={sr0:.2f}) -> {'PASS' if dsr >= 0.95 else 'FAIL'} (soglia 0.95)") + + if np.isfinite(bt["sh_f"]) and bt["sh_f"] >= 0.5: + print("\nSharpe >= 0.5 -> marginal_vs_tp01:") + m = al.marginal_vs_tp01(bt["port"]) + print(f" verdict={m.get('marginal_verdict')} corr_full={m.get('corr_full')} " + f"uplift w25 full={m['blends']['w25']['uplift_full']} hold={m['blends']['w25']['uplift_hold']} " + f"has_insample_edge={m.get('has_insample_edge')} is_hedge={m.get('is_hedge')} " + f"robust_oos={m.get('robust_oos')}") + else: + print(f"\nSharpe full {bt['sh_f']:.2f} < 0.5 -> marginal_vs_tp01 NON eseguito (sotto soglia).") + + print(f"\n[runtime {time.time()-t0:.0f}s]") + + +if __name__ == "__main__": + main() diff --git a/scripts/research/r0702_eventclock.py b/scripts/research/r0702_eventclock.py new file mode 100644 index 0000000..af33cd8 --- /dev/null +++ b/scripts/research/r0702_eventclock.py @@ -0,0 +1,453 @@ +"""r0702_eventclock.py — EVENT-CLOCK BARS (campionamento a tempo-informazione), 2026-07-02. + +IPOTESI: campionare il tempo per INFORMAZIONE (volume bars, vol bars = cum|logret|, range +bars) normalizza i regimi e migliora trend/breakout A PARITA' di strategia e frequenza media +rispetto alle barre wall-clock. Mai testato nel progetto (tutte le 104 famiglie girano su +barre wall-clock). + +DISEGNO ONESTO: + * Barre-evento costruite dal 5m certificato Deribit (al.get). Soglia CAUSALE: EWMA-90g + dell'incremento per barra 5m, SHIFTATA di 1 (solo passato), x N_target barre 5m per la + durata nominale (4h/12h/24h). Nessuna calibrazione full-sample. Parametri fissati a + priori (span 90g, warm-up 14g, durate 4/12/24h). + * Decisione a close della barra-evento k -> posizione tenuta DALLA prima barra 5m dopo la + chiusura (shift +1 barra-evento). Mark-to-market sul 5m, compounding a griglia daily + UTC (stessa convenzione di al.candidate_daily). Fee 0.0005/lato su |Δpos|. + * Selezione cella SOLO IN-SAMPLE (pre-2025) sul Sharpe 50/50; hold-out riportato per + QUELLA cella. deflated_sharpe su TUTTI i trial (event + wall). + * CONTROLLO DECISIVO: stessa strategia, stessi parametri (in unita' di barre, convertiti + per durata nominale) su barre WALL-CLOCK 4h/12h/1d (al.get, path resample leak-free). + * Guardia causalita': ricostruzione barre+target su prefisso (80%/92%) -> i confini e i + target devono coincidere con la run full troncata. + * NIENTE ffill mixed-timeframe; niente DatetimeIndex.view('int64') (uso la colonna + timestamp in ms). + +Run: uv run python scripts/research/r0702_eventclock.py +""" +from __future__ import annotations + +import sys +import time + +import numpy as np +import pandas as pd + +sys.path.insert(0, "/opt/docker/PythagorasGoal/scripts/research/alt") +import altlib as al # noqa: E402 + +FEE = al.FEE_SIDE +HOLDOUT = al.HOLDOUT +ASSETS = ("BTC", "ETH") + +# ---- parametri FISSATI A PRIORI (nessun tuning) ----------------------------------------- +DUR_HOURS = (4.0, 12.0, 24.0) # durata media nominale delle barre-evento +BAR_TYPES = ("volume", "volbar", "range") +WALL_TF = {4.0: "4h", 12.0: "12h", 24.0: "1d"} +EWM_SPAN_5M = 90 * 288 # soglia adattiva: EWMA 90 giorni di barre 5m +WARMUP_5M = 14 * 288 # min_periods 14 giorni prima della prima barra +BARS_5M_PER_H = 12 + +# strategie (parametri in GIORNI-equivalenti, convertiti in barre per durata nominale) +STRATS = [ + ("TSMOM-30/90/180", "tsmom", dict(days=(30, 90, 180))), + ("DONCH-10d", "donch", dict(days=10)), + ("DONCH-30d", "donch", dict(days=30)), + ("EWMA-5/30", "ewma", dict(days=(5, 30))), + ("EWMA-15/75", "ewma", dict(days=(15, 75))), +] + + +def bars_for_days(days: float, dur_h: float) -> int: + return max(2, int(round(days * 24.0 / dur_h))) + + +# ========================================================================================== +# COSTRUZIONE BARRE-EVENTO (causale) +# ========================================================================================== +def _increments(df5: pd.DataFrame, kind: str) -> np.ndarray: + c = df5["close"].values.astype(float) + if kind == "volume": + return df5["volume"].values.astype(float) + if kind == "volbar": # cum |logret| + r = np.zeros(len(c)); r[1:] = np.abs(np.log(c[1:] / c[:-1])) + return r + if kind == "range": # cum range relativo (high-low)/close + h = df5["high"].values.astype(float); l = df5["low"].values.astype(float) + return (h - l) / np.where(c > 0, c, np.nan) + raise ValueError(kind) + + +def _bar_close_indices(x: np.ndarray, thr: np.ndarray) -> np.ndarray: + """Loop di formazione barre: chiude una barra quando il cum degli incrementi dal + close della barra precedente raggiunge la soglia CAUSALE thr[i] (gia' shiftata).""" + e = [] + cum = 0.0 + ap = e.append + for i in range(len(x)): + t = thr[i] + if not (t > 0.0): # NaN o <=0 (warm-up): non accumulare + cum = 0.0 + continue + cum += x[i] + if cum >= t: + ap(i) + cum = 0.0 + return np.asarray(e, dtype=np.int64) + + +class EventBars: + """Barre-evento per (asset, tipo, durata): OHLC + indici 5m di chiusura.""" + + def __init__(self, df5: pd.DataFrame, kind: str, dur_h: float): + x = np.nan_to_num(_increments(df5, kind), nan=0.0) + # soglia causale: EWMA(span 90g) dell'incremento per 5m, shift(1), x N barre target + m = pd.Series(x).ewm(span=EWM_SPAN_5M, min_periods=WARMUP_5M, + adjust=False).mean().shift(1).values + n_target = dur_h * BARS_5M_PER_H + thr = m * n_target + self.e = _bar_close_indices(x, thr) # indici 5m dei close di barra + if len(self.e) < 300: + raise RuntimeError(f"troppo poche barre-evento ({len(self.e)}) per {kind}/{dur_h}h") + c5 = df5["close"].values.astype(float) + h5 = df5["high"].values.astype(float) + l5 = df5["low"].values.astype(float) + i0 = int(np.argmax(thr > 0)) # primo indice utilizzabile + starts = np.concatenate([[i0], self.e[:-1] + 1]) + sl = slice(0, self.e[-1] + 1) + self.close = c5[self.e] + self.high = np.maximum.reduceat(h5[sl], starts) + self.low = np.minimum.reduceat(l5[sl], starts) + # close-time in ms (fine barra 5m = open label + 5m); NIENTE .view su tz-aware + ts5 = df5["timestamp"].values.astype(np.int64) + self.ts_close_ms = ts5[self.e] + 300_000 + self.n5 = len(df5) + # statistiche durata + d_h = np.diff(self.ts_close_ms) / 3.6e6 + self.dur_median_h = float(np.median(d_h)) + self.dur_p5_h = float(np.percentile(d_h, 5)) + span_days = (self.ts_close_ms[-1] - self.ts_close_ms[0]) / 86.4e6 + self.bars_per_day = len(self.e) / max(span_days, 1.0) + ho_ms = int(HOLDOUT.value // 1_000_000) + mask_h = self.ts_close_ms >= ho_ms + span_h = (self.ts_close_ms[-1] - ho_ms) / 86.4e6 + self.bars_per_day_holdout = float(mask_h.sum() / max(span_h, 1.0)) + + +# ========================================================================================== +# STRATEGIE (target causale su barre-evento O wall-clock: array close/high/low) +# ========================================================================================== +def strat_target(close: np.ndarray, high: np.ndarray, low: np.ndarray, + fn: str, params: dict, dur_h: float) -> np.ndarray: + n = len(close) + if fn == "tsmom": + hs = [bars_for_days(d, dur_h) for d in params["days"]] + d = np.zeros(n) + for k in hs: + s = np.zeros(n); s[k:] = np.sign(close[k:] - close[:-k]) + d += s + t = (d > 0).astype(float) + t[:max(hs)] = 0.0 # tutte le finestre disponibili + return t + if fn == "donch": + N = bars_for_days(params["days"], dur_h) + hi = pd.Series(high).rolling(N, min_periods=N).max().shift(1).values + lo = pd.Series(low).rolling(N, min_periods=N).min().shift(1).values + pos = np.where(close > hi, 1.0, np.nan) + pos = np.where(close < lo, 0.0, pos) + return pd.Series(pos).ffill().fillna(0.0).values + if fn == "ewma": + f_d, s_d = params["days"] + fs, ss = bars_for_days(f_d, dur_h), bars_for_days(s_d, dur_h) + f = pd.Series(close).ewm(span=fs, adjust=False).mean().values + s = pd.Series(close).ewm(span=ss, adjust=False).mean().values + t = (f > s).astype(float) + t[:ss] = 0.0 + return t + raise ValueError(fn) + + +# ========================================================================================== +# VALUTAZIONE — barre-evento marked-to-market sul 5m, compounding daily +# ========================================================================================== +def pos5_from_event(n5: int, e: np.ndarray, tgt: np.ndarray) -> np.ndarray: + """Espande i target di barra-evento a posizione per-barra-5m. Il target deciso al + close della barra-evento k (indice 5m e[k]) e' tenuto DURANTE le barre 5m + (e[k], e[k+1]] -> shift +1 barra-evento by construction.""" + tgt = np.nan_to_num(np.asarray(tgt, float), nan=0.0) + pos = np.zeros(n5) + if len(e) >= 2: + pos[e[0] + 1:e[-1] + 1] = np.repeat(tgt[:-1], np.diff(e)) + if len(e) >= 1: + pos[e[-1] + 1:] = tgt[-1] + return pos + + +def daily_from_pos5(df5: pd.DataFrame, pos5: np.ndarray, fee_side: float = FEE) -> pd.Series: + c = df5["close"].values.astype(float) + r = np.zeros(len(c)); r[1:] = c[1:] / c[:-1] - 1.0 + turn = np.abs(np.diff(pos5, prepend=0.0)) + net = pos5 * r - fee_side * turn + net[0] = 0.0 + idx = pd.DatetimeIndex(pd.to_datetime(df5["datetime"], utc=True)) + return al._to_daily(pd.Series(net, index=idx)) + + +def daily_wall(asset: str, tf: str, fn: str, params: dict, dur_h: float, + fee_side: float = FEE) -> pd.Series: + df = al.get(asset, tf) + tgt = strat_target(df["close"].values.astype(float), df["high"].values.astype(float), + df["low"].values.astype(float), fn, params, dur_h) + ev = al.eval_weights(df, tgt, fee_side=fee_side) # shift +1 fatto dall'harness + return al._to_daily(pd.Series(ev["net"], index=ev["idx"])) + + +def combo5050(dA: pd.Series, dB: pd.Series) -> pd.Series: + J = pd.concat({"A": dA, "B": dB}, axis=1, join="inner").fillna(0.0) + return 0.5 * J["A"] + 0.5 * J["B"] + + +def met(d: pd.Series) -> dict: + """Sharpe/CAGR/maxDD full + hold + in-sample da una serie daily.""" + di = d[d.index < HOLDOUT]; dh = d[d.index >= HOLDOUT] + + def _cagr(s): + if len(s) < 10: + return float("nan") + tot = float(np.prod(1.0 + s.values)) + return tot ** (365.25 / len(s)) - 1.0 if tot > 0 else -1.0 + + return dict(is_sh=round(al._sh(di), 3), full_sh=round(al._sh(d), 3), + hold_sh=round(al._sh(dh), 3), full_dd=round(al._dd_ret(d), 4), + hold_dd=round(al._dd_ret(dh), 4), full_cagr=round(_cagr(d), 4), + hold_cagr=round(_cagr(dh), 4)) + + +def yearly(d: pd.Series) -> dict: + out = {} + for y, g in d.groupby(d.index.year): + eq = np.cumprod(1 + g.values); pk = np.maximum.accumulate(eq) + out[int(y)] = (round(float(eq[-1] - 1), 4), round(float(np.max((pk - eq) / pk)), 4)) + return out + + +# ========================================================================================== +# GUARDIA CAUSALITA' — ricostruzione su prefisso +# ========================================================================================== +def causality_prefix_check(asset: str, kind: str, dur_h: float, fn: str, params: dict) -> dict: + """Ricostruisce barre+target sul prefisso 80%/92% del 5m: i confini di barra devono + essere un prefisso esatto di quelli full (tranne l'ultima barra incompleta) e i target + delle barre condivise identici. Qualunque dipendenza dal futuro diverge.""" + df5 = al.get(asset, "5m") + full = EventBars(df5, kind, dur_h) + t_full = strat_target(full.close, full.high, full.low, fn, params, dur_h) + worst = 0.0; ok = True; checked = 0 + for frac in (0.80, 0.92): + cut = int(len(df5) * frac) + sub = df5.iloc[:cut].reset_index(drop=True) + eb = EventBars(sub, kind, dur_h) + m = len(eb.e) + if not np.array_equal(eb.e, full.e[:m]): + ok = False + continue + t_sub = strat_target(eb.close, eb.high, eb.low, fn, params, dur_h) + d = float(np.max(np.abs(t_sub - t_full[:m]))) if m else 0.0 + worst = max(worst, d) + checked += 1 + return dict(ok=bool(ok and worst <= 1e-9), max_diff=worst, checked=checked) + + +# ========================================================================================== +# SMALL-CAP a $600 sulle transizioni della cella scelta +# ========================================================================================== +def smallcap_event(df5: pd.DataFrame, pos5: np.ndarray, capital=600.0, min_order=5.0) -> dict: + tgt = np.nan_to_num(pos5, nan=0.0) + held = np.empty(len(tgt)); cur = 0.0; n_tr = 0 + for i in range(len(tgt)): + if abs(tgt[i] - cur) * capital >= min_order: + cur = tgt[i]; n_tr += 1 + held[i] = cur + d_real = daily_from_pos5(df5, held) + d_mod = daily_from_pos5(df5, tgt) + return dict(realistic_sh=round(al._sh(d_real), 3), modeled_sh=round(al._sh(d_mod), 3), + haircut=round(al._sh(d_mod) - al._sh(d_real), 3), n_executed=n_tr) + + +# ========================================================================================== +# MAIN +# ========================================================================================== +def main(): + t0 = time.time() + print("=" * 100) + print("R0702 EVENT-CLOCK BARS — volume/volbar/range vs wall-clock, selezione in-sample") + print("=" * 100) + + df5 = {a: al.get(a, "5m") for a in ASSETS} + for a in ASSETS: + print(f"{a} 5m: {len(df5[a])} barre, {df5[a]['datetime'].iloc[0]} -> " + f"{df5[a]['datetime'].iloc[-1]}") + + # ---- 1. costruzione barre-evento (cache) -------------------------------------------- + print("\n--- CALIBRAZIONE CLOCK (barre/giorno; target 4h=6, 12h=2, 24h=1) ---") + bars = {} + print(f"{'asset':5s} {'tipo':7s} {'dur':>5s} {'n_bars':>7s} {'bars/g':>7s} " + f"{'bars/g HOLD':>11s} {'med(h)':>7s} {'p5(h)':>6s}") + for a in ASSETS: + for k in BAR_TYPES: + for dh in DUR_HOURS: + eb = EventBars(df5[a], k, dh) + bars[(a, k, dh)] = eb + print(f"{a:5s} {k:7s} {dh:4.0f}h {len(eb.e):7d} {eb.bars_per_day:7.2f} " + f"{eb.bars_per_day_holdout:11.2f} {eb.dur_median_h:7.2f} {eb.dur_p5_h:6.2f}") + + # ---- 2. tutte le celle: event (3 tipi x 3 durate x 5 strategie) + wall (3 tf x 5) --- + cells = [] # dict(kind, bar_type, dur_h, strat, daily {asset}, daily5050, met) + for sname, fn, params in STRATS: + for dh in DUR_HOURS: + # wall-clock control + dw = {a: daily_wall(a, WALL_TF[dh], fn, params, dh) for a in ASSETS} + c5050 = combo5050(dw["BTC"], dw["ETH"]) + cells.append(dict(kind="wall", bar_type=WALL_TF[dh], dur_h=dh, strat=sname, + fn=fn, params=params, daily=dw, d5050=c5050, met=met(c5050))) + # event-clock + for k in BAR_TYPES: + de = {} + for a in ASSETS: + eb = bars[(a, k, dh)] + tgt = strat_target(eb.close, eb.high, eb.low, fn, params, dh) + de[a] = daily_from_pos5(df5[a], pos5_from_event(eb.n5, eb.e, tgt)) + c5050 = combo5050(de["BTC"], de["ETH"]) + cells.append(dict(kind="event", bar_type=k, dur_h=dh, strat=sname, + fn=fn, params=params, daily=de, d5050=c5050, met=met(c5050))) + + print(f"\n--- TUTTE LE CELLE (Sharpe 50/50: IN-SAMPLE pre-2025 | FULL | HOLD 2025-26) ---") + print(f"{'clock':6s} {'barre':7s} {'dur':>4s} {'strategia':16s} {'IS':>6s} {'FULL':>6s} {'HOLD':>6s}") + for c in sorted(cells, key=lambda x: (x["strat"], x["dur_h"], x["kind"], x["bar_type"])): + m = c["met"] + print(f"{c['kind']:6s} {c['bar_type']:7s} {c['dur_h']:3.0f}h {c['strat']:16s} " + f"{m['is_sh']:6.2f} {m['full_sh']:6.2f} {m['hold_sh']:6.2f}") + + # ---- 3. CONTROLLO DECISIVO: paired event vs wall a parita' di strategia+durata ------ + print("\n--- PAIRED: event vs wall (Δ Sharpe = event − wall, per cella accoppiata) ---") + print(f"{'strategia':16s} {'dur':>4s} {'tipo':7s} {'ΔIS':>7s} {'ΔHOLD':>7s}") + n_pairs = n_is_win = n_hold_win = n_both_win = 0 + for sname, fn, params in STRATS: + for dh in DUR_HOURS: + w = next(c for c in cells if c["kind"] == "wall" and c["strat"] == sname + and c["dur_h"] == dh) + for k in BAR_TYPES: + e = next(c for c in cells if c["kind"] == "event" and c["strat"] == sname + and c["dur_h"] == dh and c["bar_type"] == k) + d_is = e["met"]["is_sh"] - w["met"]["is_sh"] + d_h = e["met"]["hold_sh"] - w["met"]["hold_sh"] + n_pairs += 1 + n_is_win += d_is > 0 + n_hold_win += d_h > 0 + n_both_win += (d_is > 0 and d_h > 0) + print(f"{sname:16s} {dh:3.0f}h {k:7s} {d_is:+7.2f} {d_h:+7.2f}") + print(f"\nevent batte wall: IS {n_is_win}/{n_pairs}, HOLD {n_hold_win}/{n_pairs}, " + f"ENTRAMBI {n_both_win}/{n_pairs}") + + # ---- 4. selezione IN-SAMPLE della cella event migliore ------------------------------ + ev_cells = [c for c in cells if c["kind"] == "event"] + wall_cells = [c for c in cells if c["kind"] == "wall"] + chosen = max(ev_cells, key=lambda c: c["met"]["is_sh"]) + paired = next(c for c in wall_cells if c["strat"] == chosen["strat"] + and c["dur_h"] == chosen["dur_h"]) + best_wall_is = max(wall_cells, key=lambda c: c["met"]["is_sh"]) + + print("\n" + "=" * 100) + print(f"CELLA SCELTA (max Sharpe IN-SAMPLE 50/50 tra le {len(ev_cells)} event): " + f"{chosen['bar_type']} {chosen['dur_h']:.0f}h {chosen['strat']}") + print("=" * 100) + for label, d in (("BTC", chosen["daily"]["BTC"]), ("ETH", chosen["daily"]["ETH"]), + ("50/50", chosen["d5050"])): + m = met(d) + print(f" {label:6s} FULL Sh {m['full_sh']:+.2f} DD {m['full_dd']*100:5.1f}% " + f"CAGR {m['full_cagr']*100:+6.1f}% | HOLD Sh {m['hold_sh']:+.2f} " + f"DD {m['hold_dd']*100:5.1f}% CAGR {m['hold_cagr']*100:+6.1f}% | IS {m['is_sh']:+.2f}") + print(f"\n paired wall ({paired['bar_type']}, stessa strategia):") + for label, d in (("BTC", paired["daily"]["BTC"]), ("ETH", paired["daily"]["ETH"]), + ("50/50", paired["d5050"])): + m = met(d) + print(f" {label:6s} FULL Sh {m['full_sh']:+.2f} DD {m['full_dd']*100:5.1f}% " + f"CAGR {m['full_cagr']*100:+6.1f}% | HOLD Sh {m['hold_sh']:+.2f} " + f"DD {m['hold_dd']*100:5.1f}% CAGR {m['hold_cagr']*100:+6.1f}% | IS {m['is_sh']:+.2f}") + bw = best_wall_is["met"] + print(f"\n best WALL in-sample: {best_wall_is['bar_type']} {best_wall_is['strat']} " + f"IS {bw['is_sh']:+.2f} FULL {bw['full_sh']:+.2f} HOLD {bw['hold_sh']:+.2f}") + + print("\n per-anno 50/50 cella scelta (ret, maxDD):") + for y, (r, dd) in yearly(chosen["d5050"]).items(): + print(f" {y}: {r*100:+6.1f}% dd {dd*100:5.1f}%") + + # decisive control per-asset + print("\n CONTROLLO DECISIVO per-asset (event − wall):") + dec_ok = True + for a in ASSETS: + me, mw = met(chosen["daily"][a]), met(paired["daily"][a]) + d_is, d_h = me["is_sh"] - mw["is_sh"], me["hold_sh"] - mw["hold_sh"] + dec_ok = dec_ok and (d_is > 0 and d_h > 0) + print(f" {a}: ΔIS {d_is:+.2f} ΔHOLD {d_h:+.2f}") + print(f" event batte wall IS E HOLD su entrambi gli asset: {dec_ok}") + + # ---- 5. fee sweep sulla cella scelta ------------------------------------------------- + print("\n FEE SWEEP (Sharpe FULL 50/50):") + fee_sh = {} + for f in al.FEE_SWEEP: + dd_ = {} + for a in ASSETS: + eb = bars[(a, chosen["bar_type"], chosen["dur_h"])] + tgt = strat_target(eb.close, eb.high, eb.low, chosen["fn"], chosen["params"], + chosen["dur_h"]) + dd_[a] = daily_from_pos5(df5[a], pos5_from_event(eb.n5, eb.e, tgt), fee_side=f) + fee_sh[f] = round(al._sh(combo5050(dd_["BTC"], dd_["ETH"])), 3) + print(f" {2*f*100:.2f}%RT: {fee_sh[f]:+.2f}") + fee_ok = fee_sh[0.0015] > 0 + + # ---- 6. deflated Sharpe su TUTTI i trial --------------------------------------------- + all_sr = [c["met"]["full_sh"] for c in cells] + dsr, sr0 = al.deflated_sharpe(al._sh(chosen["d5050"]), all_sr, chosen["d5050"].values) + print(f"\n DEFLATED SHARPE: DSR={dsr:.3f} (soglia 0.95) | expected null max Sh {sr0:.2f} " + f"| trial totali {len(all_sr)} (event {len(ev_cells)} + wall {len(wall_cells)})") + + # ---- 7. marginal vs TP01 ------------------------------------------------------------- + print("\n MARGINAL vs TP01 (cella scelta in-sample):") + marg = al.marginal_vs_tp01(chosen["d5050"]) + for kk in ("marginal_verdict", "corr_full", "corr_hold", "cand_insample_sharpe", + "has_insample_edge", "is_hedge", "robust_oos", "multicut_uplift", + "multicut_persistent", "clean_year_uplift", "jackknife_min_uplift", + "beta_to_tp01", "resid_sharpe_full", "hedge_yearly_corr", + "uplift_tp01_up", "uplift_tp01_down"): + print(f" {kk}: {marg.get(kk)}") + for w, b in marg.get("blends", {}).items(): + print(f" blend {w}: full {b['full']} (uplift {b['uplift_full']:+.3f}) " + f"hold {b['hold']} (uplift {b['uplift_hold']:+.3f})") + earns = (marg.get("marginal_verdict") == "ADDS" and marg.get("robust_oos", False) + and marg.get("has_insample_edge", False) and not marg.get("is_hedge", False)) + dsr_pass = np.isfinite(dsr) and dsr >= 0.95 + print(f" earns_slot(marginale)={earns} dsr_pass={dsr_pass} " + f"earns_slot_honest={earns and dsr_pass and fee_ok}") + + # ---- 8. causalita' + executability ---------------------------------------------------- + print("\n GUARDIA CAUSALITA' (prefisso 80%/92%, entrambi gli asset):") + for a in ASSETS: + cz = causality_prefix_check(a, chosen["bar_type"], chosen["dur_h"], + chosen["fn"], chosen["params"]) + print(f" {a}: ok={cz['ok']} max_diff={cz['max_diff']:.2e} checked={cz['checked']}") + + print("\n EXECUTABILITY:") + for a in ASSETS: + eb = bars[(a, chosen["bar_type"], chosen["dur_h"])] + tgt = strat_target(eb.close, eb.high, eb.low, chosen["fn"], chosen["params"], + chosen["dur_h"]) + sc = smallcap_event(df5[a], pos5_from_event(eb.n5, eb.e, tgt)) + print(f" {a}: {eb.bars_per_day:.2f} barre/g (hold-out {eb.bars_per_day_holdout:.2f}), " + f"durata mediana {eb.dur_median_h:.1f}h p5 {eb.dur_p5_h:.1f}h | " + f"smallcap $600: modeled {sc['modeled_sh']:+.2f} realistic {sc['realistic_sh']:+.2f} " + f"haircut {sc['haircut']:+.3f} ({sc['n_executed']} trade)") + + print(f"\n[done in {time.time()-t0:.0f}s]") + + +if __name__ == "__main__": + main() diff --git a/scripts/research/r0702_expiry_calendar.py b/scripts/research/r0702_expiry_calendar.py new file mode 100644 index 0000000..1976666 --- /dev/null +++ b/scripts/research/r0702_expiry_calendar.py @@ -0,0 +1,450 @@ +#!/usr/bin/env python +"""r0702_expiry_calendar.py — FILONE: effetti del calendario SCADENZE Deribit (2026-07-02). + +Deribit: opzioni settimanali scadono ogni VENERDI' 08:00 UTC; mensili l'ultimo venerdi' +del mese 08:00 UTC; trimestrali l'ultimo venerdi' di mar/giu/set/dic. Ipotesi: pinning / +compressione pre-expiry, drift post-expiry (rimozione hedging dealer), pattern di vol. + +=== GRIGLIA DICHIARATA PRIMA DI GUARDARE I DATI (nessun cherry-picking a posteriori) === + Finestre evento (24h, allineate alla griglia giornaliera ancorata alle 08:00 UTC): + W-2 = [-48h,-24h) W-1 = [-24h,0) W0 = [0,+24h) W+1 = [+24h,+48h) + Tipi expiry: WEEKLY (ogni venerdi' 08:00), MONTHLY (ultimo venerdi' del mese), + QUARTERLY (ultimo venerdi' di mar/giu/set/dic). NB: MONTHLY subset di WEEKLY, + QUARTERLY subset di MONTHLY (nesting dichiarato). + Asset: BTC, ETH. => 24 celle base (3 tipi x 4 finestre x 2 asset). + Multiple testing: Bonferroni a 24 celle, alpha 5% due code -> |t| >= 3.09. + + CONFOUND STRUTTURALE dichiarato: il WEEKLY e' osservazionalmente IDENTICO al + day-of-week venerdi' (ogni venerdi' e' un expiry: non esistono venerdi' di controllo) + e SEA02 (day-of-week) e' gia' morto. Quindi: + - CONTRASTI CHIAVE separabili: MONTHLY vs ALTRI venerdi' (controlla il day-of-week), + QUARTERLY vs ALTRE monthly. Sono questi i test che possono dare un PASS. + NULL (tutti e tre, un effetto vero li passa tutti): + (a) PLACEBO WEEKDAY: ancora lun/mar/mer/gio 08:00 (weekly) e last-lun..last-gio + del mese (monthly): il venerdi'/ultimo-venerdi' deve essere speciale. + (b) ANCHOR-SHIFT: ancora a 08:00 +/-2h/+/-4h (04/06/08/10/12): un evento reale + degrada gradualmente, un artefatto di etichettatura si inverte. + (c) PERMUTATION: 500 calendari con stesso n eventi/anno. + perm-A = giorni casuali (qualsiasi weekday); perm-B (piu' affilato) = + venerdi' casuali (monthly) / ultimi-venerdi'-del-mese casuali (quarterly). + Statistica prima della strategia. Regola tradabile costruita SOLO se una cella passa: + |t2|>=3.09 (Bonferroni) AND placebo AND anchor-shift senza inversione AND perm pctl + estremo (<=1% o >=99%). La famiglia strategica (3 tipi x 4 finestre x 2 direzioni = + 24 trial) viene comunque valutata per riportare deflated_sharpe con n trial onesto. + +Vincoli rispettati: nessun .view("int64") su datetimes tz-aware (epoca esplicita in ms +via colonna `timestamp` gia' in ms); posizioni causali (target[i] deciso a close[i], +tenuto nella barra i+1 — eval_weights shifta); fee 0.10% RT. +""" +from __future__ import annotations + +import sys + +sys.path.insert(0, "/opt/docker/PythagorasGoal/scripts/research/alt") +import numpy as np +import pandas as pd +from scipy import stats as sps + +import altlib as al + +RNG_SEED = 20260702 +N_PERM = 500 +ANCHOR_HOUR = 8 +OFFSETS = {"[-48,-24)": -2, "[-24,0)": -1, "[0,+24)": 0, "[+24,+48)": +1} +N_CELLS = 24 # 3 tipi x 4 finestre x 2 asset — dichiarato +BONF_T = float(sps.norm.ppf(1 - 0.025 / N_CELLS)) # ~3.09 +ASSETS = ("BTC", "ETH") +ETYPES = ("WEEKLY", "MONTHLY", "QUARTERLY") +MS_H = 3_600_000 +MS_D = 24 * MS_H + + +# =========================================================================== +# CALENDARIO SCADENZE (funzione pura, nessun dato di mercato) +# =========================================================================== +def _utc_index(values) -> pd.DatetimeIndex: + idx = pd.DatetimeIndex(values) + return idx.tz_localize("UTC") if idx.tz is None else idx.tz_convert("UTC") + + +def expiry_calendar(start: str, end: str, anchor_hour: int = ANCHOR_HOUR) -> dict: + """Ancore expiry Deribit (tz UTC, ore = anchor_hour). Ritorna dict tipo->DatetimeIndex.""" + days = pd.date_range(start, end, freq="D", tz="UTC") + fri = days[days.weekday == 4] + weekly = pd.DatetimeIndex(fri) + pd.Timedelta(hours=anchor_hour) + ym = np.asarray(fri.year * 100 + fri.month) + per = pd.Series(fri).groupby(ym).max() + monthly = _utc_index(per.to_numpy()) + pd.Timedelta(hours=anchor_hour) + quarterly = monthly[monthly.month.isin([3, 6, 9, 12])] + return {"WEEKLY": weekly, "MONTHLY": monthly, "QUARTERLY": quarterly} + + +def placebo_calendar(start: str, end: str, weekday: int, anchor_hour: int = ANCHOR_HOUR) -> dict: + """Placebo: stesso costrutto ancorato a un ALTRO giorno della settimana.""" + days = pd.date_range(start, end, freq="D", tz="UTC") + wd = days[days.weekday == weekday] + weekly = pd.DatetimeIndex(wd) + pd.Timedelta(hours=anchor_hour) + ym = np.asarray(wd.year * 100 + wd.month) + per = pd.Series(wd).groupby(ym).max() + monthly = _utc_index(per.to_numpy()) + pd.Timedelta(hours=anchor_hour) + return {"WEEKLY": weekly, "MONTHLY": monthly} + + +# =========================================================================== +# GRIGLIA GIORNALIERA ancorata (ritorno log 24h + RV) — tutta epoca ms esplicita +# =========================================================================== +def day_table(asset: str, anchor_hour: int = ANCHOR_HOUR) -> pd.DataFrame: + """Partiziona le barre 1h in 'giorni' [anchor, anchor+24h). Ritorna per giorno: + ret (somma log-ret orari = log-ret close->close della finestra), rv (std oraria + annualizzata), n barre. r[i] copre ~[open_i, open_i+1h) => giorno = open in finestra.""" + df = al.get(asset, "1h") + ts = df["timestamp"].to_numpy(dtype=np.int64) # epoca ms esplicita + c = df["close"].to_numpy(dtype=float) + lr = np.zeros(len(c)) + lr[1:] = np.log(c[1:] / c[:-1]) + day_ms = ((ts - anchor_hour * MS_H) // MS_D) * MS_D + anchor_hour * MS_H + g = pd.DataFrame({"day_ms": day_ms, "lr": lr}) + agg = g.groupby("day_ms")["lr"].agg(ret="sum", rv="std", n="size") + agg = agg[agg["n"] >= 20] # solo giorni ~completi + agg["rv"] = agg["rv"] * np.sqrt(24 * 365.25) # RV annualizzata + agg.index = pd.to_datetime(agg.index, unit="ms", utc=True) + return agg + + +_EPOCH = pd.Timestamp("1970-01-01", tz="UTC") + + +def anchors_ms(anchors: pd.DatetimeIndex) -> np.ndarray: + """Epoca ms ESPLICITA e unit-safe: in pandas 2.x un DatetimeIndex tz-aware puo' + essere in unita' s/ms/ns (.asi8 cambia scala!) — la delta da EPOCH no.""" + delta = pd.DatetimeIndex(anchors) - _EPOCH + return np.asarray(delta // pd.Timedelta(milliseconds=1), dtype=np.int64) + + +def window_days(agg: pd.DataFrame, anchors: pd.DatetimeIndex, offset: int) -> pd.DataFrame: + """I giorni-griglia che iniziano a (anchor + offset*24h) e cadono nel campione.""" + starts = pd.DatetimeIndex(pd.to_datetime( + anchors_ms(anchors) + offset * MS_D, unit="ms", utc=True)) + return agg.loc[agg.index.isin(starts)] + + +def cell_stats(agg: pd.DataFrame, anchors: pd.DatetimeIndex, offset: int) -> dict: + ev = window_days(agg, anchors, offset) + base = agg.loc[~agg.index.isin(ev.index)] + r = ev["ret"].to_numpy() + if len(r) < 8: + return dict(n=len(r)) + mean, med = float(r.mean()), float(np.median(r)) + sem = float(r.std(ddof=1) / np.sqrt(len(r))) + t1 = mean / sem if sem > 0 else 0.0 # vs zero + t2, p2 = sps.ttest_ind(r, base["ret"].to_numpy(), equal_var=False) # vs tutte le altre finestre + rv_ev, rv_b = float(ev["rv"].mean()), float(base["rv"].mean()) + trv, prv = sps.ttest_ind(ev["rv"].dropna(), base["rv"].dropna(), equal_var=False) + ci = 1.96 * sem + return dict(n=len(r), mean=mean, median=med, ci95=ci, t1=float(t1), + t2=float(t2), p2=float(p2), base_mean=float(base["ret"].mean()), + rv_ev=rv_ev, rv_base=rv_b, rv_ratio=rv_ev / rv_b if rv_b > 0 else np.nan, + t_rv=float(trv)) + + +def contrast_stats(agg: pd.DataFrame, a1: pd.DatetimeIndex, a2: pd.DatetimeIndex, + offset: int) -> dict: + """Welch t fra finestre-evento di due calendari (es. MONTHLY vs altri venerdi').""" + e1 = window_days(agg, a1, offset)["ret"].to_numpy() + e2 = window_days(agg, a2, offset)["ret"].to_numpy() + if len(e1) < 8 or len(e2) < 8: + return dict(n1=len(e1), n2=len(e2)) + t, p = sps.ttest_ind(e1, e2, equal_var=False) + return dict(n1=len(e1), n2=len(e2), m1=float(e1.mean()), m2=float(e2.mean()), + diff=float(e1.mean() - e2.mean()), t=float(t), p=float(p)) + + +# =========================================================================== +# PERMUTATION NULL — 500 calendari, stesso n eventi/anno +# =========================================================================== +def permutation_null(agg: pd.DataFrame, anchors: pd.DatetimeIndex, offsets: dict, + pool: pd.DatetimeIndex, n_perm: int = N_PERM, + seed: int = RNG_SEED) -> dict: + """Percentile della media reale per finestra vs n_perm calendari casuali estratti da + `pool` (stesso numero di ancore per anno del calendario reale, senza rimpiazzo).""" + rng = np.random.default_rng(seed) + full_idx = pd.date_range(agg.index.min(), agg.index.max(), freq="24h") + ret_full = agg["ret"].reindex(full_idx).to_numpy() # NaN dove giorno mancante + t0 = int(anchors_ms(full_idx)[0]) + + def pos_of(idx: pd.DatetimeIndex) -> np.ndarray: + return ((anchors_ms(idx) - t0) // MS_D).astype(np.int64) + + n_days = len(full_idx) + a_pos = pos_of(anchors) + a_pos = a_pos[(a_pos >= 2) & (a_pos < n_days - 2)] + pool_pos = pos_of(pool) + pool_pos = pool_pos[(pool_pos >= 2) & (pool_pos < n_days - 2)] + years_a = pd.to_datetime((a_pos * MS_D + t0), unit="ms", utc=True).year + years_p = pd.to_datetime((pool_pos * MS_D + t0), unit="ms", utc=True).year + per_year = pd.Series(a_pos).groupby(np.asarray(years_a)).size() + pool_by_year = {y: pool_pos[years_p == y] for y in per_year.index} + + def win_means(pos: np.ndarray) -> dict: + out = {} + for wname, off in offsets.items(): + v = ret_full[pos + off] + out[wname] = float(np.nanmean(v)) + return out + + real = win_means(a_pos) + null = {w: np.empty(n_perm) for w in offsets} + for k in range(n_perm): + draw = [] + for y, cnt in per_year.items(): + p = pool_by_year.get(y, np.array([], dtype=np.int64)) + if len(p) == 0: + continue + take = min(cnt, len(p)) + draw.append(rng.choice(p, size=take, replace=False)) + pos = np.concatenate(draw) if draw else np.array([], dtype=np.int64) + wm = win_means(pos) + for w in offsets: + null[w][k] = wm[w] + return {w: dict(real=real[w], + pctl=float(np.mean(null[w] <= real[w])), + null_mean=float(np.nanmean(null[w])), + null_sd=float(np.nanstd(null[w]))) + for w in offsets} + + +# =========================================================================== +# STRATEGIA (famiglia dichiarata: 3 tipi x 4 finestre x 2 direzioni = 24 trial) +# =========================================================================== +def make_expiry_target(anchors: pd.DatetimeIndex, offset: int, direction: float): + """target[i] = direction se la PROSSIMA barra (open+1h) cade nella finestra + [anchor+offset*24h, anchor+(offset+1)*24h). Calendario noto ex-ante => causale; + eval_weights shifta comunque di +1 barra (decidi a close[i], agisci in i+1).""" + ws = np.sort(anchors_ms(anchors) + offset * MS_D) # start finestre, ms + + def target_fn(df: pd.DataFrame) -> np.ndarray: + ts = df["timestamp"].to_numpy(dtype=np.int64) + MS_H # open prossima barra + j = np.searchsorted(ws, ts, side="right") - 1 + ok = (j >= 0) & ((ts - ws[np.clip(j, 0, len(ws) - 1)]) < MS_D) + return np.where(ok, direction, 0.0) + + return target_fn + + +def strategy_family(cal: dict) -> list[dict]: + fam = [] + for et in ETYPES: + for wname, off in OFFSETS.items(): + for d in (+1.0, -1.0): + fam.append(dict(etype=et, window=wname, offset=off, direction=d, + fn=make_expiry_target(cal[et], off, d))) + return fam + + +# =========================================================================== +# MAIN +# =========================================================================== +def main() -> None: + aggs = {a: day_table(a) for a in ASSETS} + spans = {a: (aggs[a].index.min(), aggs[a].index.max()) for a in ASSETS} + cal_start = min(s[0] for s in spans.values()) - pd.Timedelta(days=3) + cal_end = max(s[1] for s in spans.values()) + pd.Timedelta(days=3) + cal = expiry_calendar(str(cal_start.date()), str(cal_end.date())) + print(f"Span dati (griglia 08:00): " + + "; ".join(f"{a} {spans[a][0].date()}->{spans[a][1].date()} ({len(aggs[a])}g)" + for a in ASSETS)) + print(f"Eventi calendario: " + ", ".join(f"{k}={len(v)}" for k, v in cal.items())) + print(f"Soglia Bonferroni (24 celle, 5% due code): |t2| >= {BONF_T:.2f}\n") + + # ------------------------------------------------------------------ (3) statistica + print("=" * 100) + print("(1) EFFETTI PER FINESTRA x TIPO-EXPIRY x ASSET — ret log 24h; t1 vs 0, t2 vs TUTTE le altre finestre") + print("=" * 100) + cells = {} + for a in ASSETS: + for et in ETYPES: + for wname, off in OFFSETS.items(): + st = cell_stats(aggs[a], cal[et], off) + cells[(a, et, wname)] = st + if st.get("n", 0) >= 8: + print(f"{a} {et:9s} {wname:10s} n={st['n']:4d} " + f"mean={st['mean']*100:+.3f}%±{st['ci95']*100:.3f} " + f"med={st['median']*100:+.3f}% t1={st['t1']:+.2f} t2={st['t2']:+.2f} " + f"| RV ev/base={st['rv_ev']:.3f}/{st['rv_base']:.3f} " + f"ratio={st['rv_ratio']:.2f} tRV={st['t_rv']:+.2f}") + print("-" * 100) + + print("\nPer-anno (mean ret % della finestra; n eventi):") + for a in ASSETS: + for et in ETYPES: + years = sorted(set(aggs[a].index.year)) + for wname, off in OFFSETS.items(): + ev = window_days(aggs[a], cal[et], off) + parts = [] + for y in years: + r = ev[ev.index.year == y]["ret"] + parts.append(f"{y}:{r.mean()*100:+.2f}({len(r)})" if len(r) else f"{y}:--") + print(f"{a} {et:9s} {wname:10s} " + " ".join(parts)) + print() + + # -------------------------------------------------- contrasti chiave (separabili) + print("=" * 100) + print("CONTRASTI CHIAVE (separano l'expiry dal day-of-week): MONTHLY vs ALTRI venerdi'; QUARTERLY vs ALTRE monthly") + print("=" * 100) + other_fri = cal["WEEKLY"][~cal["WEEKLY"].isin(cal["MONTHLY"])] + other_mon = cal["MONTHLY"][~cal["MONTHLY"].isin(cal["QUARTERLY"])] + contrasts = {} + for a in ASSETS: + for label, (a1, a2) in {"MONTHLY-vs-otherFRI": (cal["MONTHLY"], other_fri), + "QUARTERLY-vs-otherMON": (cal["QUARTERLY"], other_mon)}.items(): + for wname, off in OFFSETS.items(): + cs = contrast_stats(aggs[a], a1, a2, off) + contrasts[(a, label, wname)] = cs + if cs.get("n1", 0) >= 8: + print(f"{a} {label:22s} {wname:10s} n={cs['n1']}/{cs['n2']} " + f"m_ev={cs['m1']*100:+.3f}% m_ctrl={cs['m2']*100:+.3f}% " + f"diff={cs['diff']*100:+.3f}% t={cs['t']:+.2f} p={cs['p']:.3f}") + + # ------------------------------------------------------------------ (4a) placebo + print("\n" + "=" * 100) + print("(2a) PLACEBO WEEKDAY — stesso costrutto ancorato a lun/mar/mer/gio (t2 vs base; venerdi' deve spiccare)") + print("=" * 100) + wd_names = {0: "MON", 1: "TUE", 2: "WED", 3: "THU", 4: "FRI(reale)"} + placebo_t2 = {} + for a in ASSETS: + for et in ("WEEKLY", "MONTHLY"): + for wname, off in OFFSETS.items(): + row = {} + for wd in (0, 1, 2, 3): + pc = placebo_calendar(str(cal_start.date()), str(cal_end.date()), wd) + st = cell_stats(aggs[a], pc[et], off) + row[wd_names[wd]] = st.get("t2", np.nan) + row[wd_names[4]] = cells[(a, et, wname)].get("t2", np.nan) + placebo_t2[(a, et, wname)] = row + print(f"{a} {et:8s} {wname:10s} " + + " ".join(f"{k}:{v:+.2f}" for k, v in row.items())) + + # -------------------------------------------------------------- (4b) anchor-shift + print("\n" + "=" * 100) + print("(2b) ANCHOR-SHIFT — media evento (%) con ancora a 04/06/08/10/12 UTC (reale=08). Inversione => artefatto") + print("=" * 100) + shift_means = {} + for a in ASSETS: + tabs = {h: day_table(a, anchor_hour=h) for h in (4, 6, 8, 10, 12)} + for et in ETYPES: + for wname, off in OFFSETS.items(): + row = {} + for h in (4, 6, 8, 10, 12): + calh = expiry_calendar(str(cal_start.date()), str(cal_end.date()), + anchor_hour=h) + st = cell_stats(tabs[h], calh[et], off) + row[h] = st.get("mean", np.nan) + shift_means[(a, et, wname)] = row + base = row[8] + vals = [row[h] for h in (4, 6, 10, 12) if np.isfinite(row.get(h, np.nan))] + inverts = (np.isfinite(base) and abs(base) > 0 and + any(np.sign(v) == -np.sign(base) and abs(v) > 0.5 * abs(base) + for v in vals)) + print(f"{a} {et:9s} {wname:10s} " + + " ".join(f"{h:02d}h:{row[h]*100:+.3f}%" for h in (4, 6, 8, 10, 12)) + + f" inverts={inverts}") + + # -------------------------------------------------------------- (4c) permutation + print("\n" + "=" * 100) + print(f"(2c) PERMUTATION NULL — {N_PERM} calendari, stesso n eventi/anno. pctl = quota null <= reale") + print(" perm-A: giorni casuali. perm-B (affilato): venerdi' casuali (MONTHLY) / monthly casuali (QUARTERLY)") + print("=" * 100) + perm = {} + for a in ASSETS: + idx_all = aggs[a].index + pool_any = idx_all + pool_fri = idx_all[idx_all.weekday == 4] + pool_mon_expiry = idx_all[idx_all.isin(cal["MONTHLY"])] + specs = {("WEEKLY", "A"): (cal["WEEKLY"], pool_any), + ("MONTHLY", "A"): (cal["MONTHLY"], pool_any), + ("MONTHLY", "B"): (cal["MONTHLY"], pool_fri), + ("QUARTERLY", "A"): (cal["QUARTERLY"], pool_any), + ("QUARTERLY", "B"): (cal["QUARTERLY"], pool_mon_expiry)} + for (et, mode), (anch, pool) in specs.items(): + res = permutation_null(aggs[a], anch, OFFSETS, pool) + for wname, r in res.items(): + perm[(a, et, mode, wname)] = r + print(f"{a} {et:9s} perm-{mode} " + + " ".join(f"{w}:{r['pctl']*100:.1f}%" for w, r in res.items())) + + # ------------------------------------------------------- gate statistico dichiarato + print("\n" + "=" * 100) + print("GATE (dichiarato in testa): |t2|>=Bonferroni AND placebo AND no-inversione AND perm pctl <=1% o >=99%") + print("=" * 100) + survivors = [] + for a in ASSETS: + for et in ETYPES: + for wname in OFFSETS: + st = cells[(a, et, wname)] + if st.get("n", 0) < 8: + continue + t2 = st["t2"] + bonf_ok = abs(t2) >= BONF_T + pt = placebo_t2.get((a, et, wname)) + placebo_ok = (pt is None or + abs(pt["FRI(reale)"]) > max(abs(pt[k]) for k in + ("MON", "TUE", "WED", "THU"))) + row = shift_means[(a, et, wname)] + base = row[8] + shift_ok = not (np.isfinite(base) and any( + np.isfinite(row[h]) and np.sign(row[h]) == -np.sign(base) + and abs(row[h]) > 0.5 * abs(base) for h in (4, 6, 10, 12))) + pA = perm.get((a, et, "A", wname), {}).get("pctl", 0.5) + pB = perm.get((a, et, "B", wname), {}).get("pctl", pA) + perm_ok = all(p <= 0.01 or p >= 0.99 for p in (pA, pB)) + ok = bonf_ok and placebo_ok and shift_ok and perm_ok + flag = " <== SURVIVES" if ok else "" + print(f"{a} {et:9s} {wname:10s} t2={t2:+.2f} bonf={bonf_ok} " + f"placebo={placebo_ok} shift_ok={shift_ok} " + f"permA={pA:.3f} permB={pB:.3f} perm_ok={perm_ok}{flag}") + if ok: + survivors.append((a, et, wname, st)) + + # ---------------------------------------------- (5) famiglia strategica + DSR onesto + print("\n" + "=" * 100) + print("(3) FAMIGLIA STRATEGICA (24 trial dichiarati: 3 tipi x 4 finestre x 2 dir) — Sharpe 50/50 netto fee") + print(" Riportata SEMPRE per il conteggio trial/DSR; study_marginal SOLO se la statistica sopravvive.") + print("=" * 100) + fam = strategy_family(cal) + rows = [] + for f in fam: + daily = al.candidate_daily(f["fn"], tf="1h") + ins = daily[daily.index < al.HOLDOUT] + rows.append(dict(etype=f["etype"], window=f["window"], direction=f["direction"], + fn=f["fn"], daily=daily, + full_sh=al._sh(daily), ins_sh=al._sh(ins), + hold_sh=al._sh(daily[daily.index >= al.HOLDOUT]))) + rows.sort(key=lambda r: r["ins_sh"], reverse=True) + for r in rows: + print(f"{r['etype']:9s} {r['window']:10s} dir={r['direction']:+.0f} " + f"insample={r['ins_sh']:+.2f} full={r['full_sh']:+.2f} hold={r['hold_sh']:+.2f}") + all_full = [r["full_sh"] for r in rows] + best = rows[0] # scelto IN-SAMPLE-ONLY (no hold-out) + dsr, sr0 = al.deflated_sharpe(best["full_sh"], all_full, best["daily"]) + print(f"\nCella best IN-SAMPLE: {best['etype']} {best['window']} dir={best['direction']:+.0f} " + f"(ins {best['ins_sh']:+.2f}, full {best['full_sh']:+.2f}, hold {best['hold_sh']:+.2f})") + print(f"deflated_sharpe (N={len(all_full)} trial): DSR={dsr:.3f} " + f"(null max atteso ~{sr0:.2f} ann.) PASS>=0.95: {dsr >= 0.95}") + + if survivors: + print("\nStatistica SOPRAVVISSUTA -> study_marginal sulla regola piu' semplice della cella best in-sample:") + rep = al.study_marginal( + f"EXPIRY-{best['etype']}-{best['window']}-d{best['direction']:+.0f}", + best["fn"], tf="1h") + print(al.fmt_marginal(rep)) + yr = al.eval_weights(al.get("BTC", "1h"), + best["fn"](al.get("BTC", "1h")))["yearly"] + print("Per-anno BTC:", {y: v["ret"] for y, v in yr.items()}) + else: + print("\nNESSUNA cella sopravvive al gate statistico -> NESSUNA regola tradabile costruita (per protocollo).") + + print(f"\nSopravvissuti al gate statistico: {len(survivors)}/24 celle") + + +if __name__ == "__main__": + main() diff --git a/scripts/research/r0702_regime_speed.py b/scripts/research/r0702_regime_speed.py new file mode 100644 index 0000000..39c5766 --- /dev/null +++ b/scripts/research/r0702_regime_speed.py @@ -0,0 +1,331 @@ +#!/usr/bin/env python +"""r0702_regime_speed — VELOCITÀ DEL TREND CONDIZIONATA DAL REGIME DI VOL (2026-07-02). + +DOMANDA: TP01 media TRE orizzonti TSMOM (30/90/180g) a PESI FISSI. Condizionare i PESI +TRA GLI ORIZZONTI (la velocità del segnale, NON la leva — l'overlay DVOL sul vol-target è +già stato scartato il 2026-06-26) al regime di volatilità migliora il fixed-blend canonico? +Ipotesi A: alta vol → trend più veloci → più peso all'orizzonte corto (hv_fast). +Ipotesi B: il contrario (hv_slow). + +METODO (onesto): + * TSMOM per orizzonte separato, long-flat, vol-target 20% / cap 2x come il canonico. + Sanity: pesi fissi 1/3-1/3-1/3 deve riprodurre il baseline TP01 (stesso code-path). + * REGIME = percentile ESPANDENTE CAUSALE (rank del valore di oggi nella storia fino a + oggi inclusa, min 365 osservazioni) di DUE misure: realized vol 30g (storia 2019+) e + DVOL Deribit (dal 2021-03, allineato causale via al.dvol / merge_asof backward su + epoca ms esplicita). Dove il percentile non è ancora definito → pesi EQUAL (canonico). + * FAMIGLIA via al.study_family_honest (selezione IN-SAMPLE + deflated Sharpe automatici): + griglia = misura {rv, dvol} × soglia {0.60, 0.75} × mappa {hv_fast, hv_slow} × + blend {hard, linear} = 16 celle (UNA famiglia sola: il DSR conta TUTTI i trial). + * ASTICELLA: il candidato è quasi-TP01 (corr ~1) → il criterio NON è earns_slot ma la + DOMINANZA del fixed-blend canonico: Sharpe FULL e HOLD >= canonico su BTC, ETH e 50/50, + uplift positivo a più date di taglio (2023/2024/2025), DSR >= 0.95. + * CONTROLLO NULL: 300 draw di PESI FISSI casuali (Dirichlet) sui 3 orizzonti — il + regime-conditioning deve battere il ~p90 del null, altrimenti è rumore di pesatura. + * Causalità: percentili espandenti (mai full-sample), eval_weights shifta la posizione, + al.causality_ok sulla cella scelta; niente .view("int64") su indici tz-aware. + +Run: uv run python scripts/research/r0702_regime_speed.py +""" +from __future__ import annotations + +import bisect +import sys + +import numpy as np +import pandas as pd + +sys.path.insert(0, "/opt/docker/PythagorasGoal/scripts/research/alt") +import altlib as al # noqa: E402 + +HORIZONS_D = (30, 90, 180) +FAST_W = np.array([0.65, 0.25, 0.10]) # tilt forte sull'orizzonte corto +SLOW_W = np.array([0.10, 0.25, 0.65]) # tilt forte sull'orizzonte lungo +EQ_W = np.array([1 / 3, 1 / 3, 1 / 3]) # canonico TP01 +MIN_REGIME_OBS = 365 # storia minima prima di fidarsi del percentile +RAMP = 0.25 # semi-larghezza del blend lineare attorno alla soglia +CUTS = ("2023-01-01", "2024-01-01", "2025-01-01") +NULL_DRAWS = 300 +SEED = 20260702 + + +# --------------------------------------------------------------------------- +# Blocchi causali +# --------------------------------------------------------------------------- +def horizon_signs(c: np.ndarray, bpd: int) -> np.ndarray: + """S[i, j] = sign(close[i]/close[i-h_j] - 1), NaN dove la storia non basta.""" + n = len(c) + S = np.full((n, len(HORIZONS_D)), np.nan) + for j, hd in enumerate(HORIZONS_D): + h = hd * bpd + if h < n: + S[h:, j] = np.sign(c[h:] / c[:-h] - 1.0) + return S + + +def direction_from_weights(S: np.ndarray, W: np.ndarray) -> np.ndarray: + """Direzione long-flat = media pesata dei sign sugli orizzonti VALIDI (pesi + rinormalizzati sui validi, come tsmom_blend rinormalizza sul conteggio).""" + V = np.isfinite(S) + Wv = np.where(V, W, 0.0) + norm = Wv.sum(axis=1) + num = (np.where(V, S, 0.0) * Wv).sum(axis=1) + d = np.where(norm > 0, num / np.where(norm > 0, norm, 1.0), 0.0) + return np.clip(d, 0.0, None) # LONG-FLAT come TP01 canonico + + +def expanding_pctl(v: np.ndarray, min_n: int = MIN_REGIME_OBS) -> np.ndarray: + """Percentile espandente CAUSALE: mid-rank di v[i] nella storia v[<=i] (NaN esclusi). + Nessuna statistica full-sample; identico ricomputato su qualunque prefisso.""" + v = np.asarray(v, float) + out = np.full(len(v), np.nan) + hist: list[float] = [] + for i in range(len(v)): + x = v[i] + if not np.isfinite(x): + continue + bisect.insort(hist, x) + if len(hist) >= min_n: + lo = bisect.bisect_left(hist, x) + hi = bisect.bisect_right(hist, x) + out[i] = (lo + hi) / 2.0 / len(hist) + return out + + +def regime_pctl(df: pd.DataFrame, asset: str, measure: str) -> np.ndarray: + bpd = al.bars_per_day(df) + if measure == "rv": + r = al.simple_returns(df["close"].values.astype(float)) + v = al.realized_vol(r, 30 * bpd, bpd * 365.25) + elif measure == "dvol": + v = al.dvol(df, asset) # merge_asof backward, epoca ms esplicita + else: + raise ValueError(measure) + return expanding_pctl(v) + + +def weight_matrix(pct: np.ndarray, thr: float, mapping: str, blend: str) -> np.ndarray: + """Pesi per barra sui 3 orizzonti. lam=1 → peso di regime ALTO, lam=0 → BASSO. + hv_fast: alto → FAST_W; hv_slow: alto → SLOW_W. hard = switch alla soglia; + linear = rampa lineare col percentile centrata sulla soglia (larghezza 2*RAMP). + Dove il percentile non è definito → EQ_W (canonico) — causale e conservativo.""" + n = len(pct) + hi_w, lo_w = (FAST_W, SLOW_W) if mapping == "hv_fast" else (SLOW_W, FAST_W) + if blend == "hard": + lam = (pct > thr).astype(float) + else: + lam = np.clip(0.5 + (pct - thr) / (2.0 * RAMP), 0.0, 1.0) + W = lam[:, None] * hi_w[None, :] + (1.0 - lam[:, None]) * lo_w[None, :] + bad = ~np.isfinite(pct) + W[bad] = EQ_W + return W + + +def make_target(thr: float, mapping: str, blend: str, measure: str): + def target(df: pd.DataFrame, asset: str) -> np.ndarray: + c = df["close"].values.astype(float) + bpd = al.bars_per_day(df) + S = horizon_signs(c, bpd) + W = weight_matrix(regime_pctl(df, asset, measure), thr, mapping, blend) + d = direction_from_weights(S, W) + return al.vol_target(d, df, 0.20, 30, 2.0) + return target + + +def fixed_target(weights: np.ndarray): + def target(df: pd.DataFrame, asset: str = "") -> np.ndarray: + c = df["close"].values.astype(float) + S = horizon_signs(c, al.bars_per_day(df)) + d = direction_from_weights(S, np.tile(weights, (len(c), 1))) + return al.vol_target(d, df, 0.20, 30, 2.0) + return target + + +def factory(tf: str = "1d", thr: float = 0.6, mapping: str = "hv_fast", + blend: str = "hard", measure: str = "rv"): + return make_target(thr, mapping, blend, measure) + + +# --------------------------------------------------------------------------- +# Valutazione: per-asset + 50/50 (stessa convenzione di candidate_daily) +# --------------------------------------------------------------------------- +def per_asset_series(target_fn) -> dict[str, pd.Series]: + out = {} + for a in al.CERTIFIED: + df = al.get(a, "1d") + ev = al.eval_weights(df, al._call_target(target_fn, df, a)) + out[a] = pd.Series(ev["net"], index=ev["idx"]) + return out + + +def combo_5050(series: dict[str, pd.Series]) -> pd.Series: + J = pd.concat(series, axis=1, join="inner").fillna(0.0) + return al._to_daily(0.5 * J[al.CERTIFIED[0]] + 0.5 * J[al.CERTIFIED[1]]) + + +def sh_full_hold(s: pd.Series) -> tuple[float, float]: + return al._sh(s), al._sh(s[s.index >= al.HOLDOUT]) + + +def dominance_table(cand: dict[str, pd.Series], ctrl: dict[str, pd.Series]) -> dict: + """Sharpe FULL/HOLD per BTC, ETH, 50/50: candidato vs controllo fixed-blend.""" + rows = {} + for k in ["BTC", "ETH", "5050"]: + cs = combo_5050(cand) if k == "5050" else al._to_daily(cand[k]) + bs = combo_5050(ctrl) if k == "5050" else al._to_daily(ctrl[k]) + cf, chd = sh_full_hold(cs) + bf, bh = sh_full_hold(bs) + rows[k] = dict(cand_full=round(cf, 3), ctrl_full=round(bf, 3), d_full=round(cf - bf, 3), + cand_hold=round(chd, 3), ctrl_hold=round(bh, 3), d_hold=round(chd - bh, 3)) + return rows + + +def multicut(cand_5050: pd.Series, ctrl_5050: pd.Series) -> dict: + out = {} + for cut in CUTS: + t = pd.Timestamp(cut, tz="UTC") + c, b = cand_5050[cand_5050.index >= t], ctrl_5050[ctrl_5050.index >= t] + out[cut] = round(al._sh(c) - al._sh(b), 3) + return out + + +def dd_of(s: pd.Series) -> float: + return round(al._dd_ret(s), 4) + + +# --------------------------------------------------------------------------- +# NULL: 300 pesi fissi casuali sui 3 orizzonti (fast path vettoriale) +# --------------------------------------------------------------------------- +def null_fixed_weights(n_draws: int = NULL_DRAWS, seed: int = SEED): + pre = {} + for a in al.CERTIFIED: + df = al.get(a, "1d") + c = df["close"].values.astype(float) + bpd = al.bars_per_day(df) + r = al.simple_returns(c) + vol = al.realized_vol(r, 30 * bpd, bpd * 365.25) + scal = np.where((vol > 0) & np.isfinite(vol), 0.20 / vol, 0.0) + idx = pd.DatetimeIndex(pd.to_datetime(df["datetime"], utc=True)) + pre[a] = dict(S=horizon_signs(c, bpd), scal=scal, r=r, idx=idx) + rng = np.random.default_rng(seed) + draws = rng.dirichlet(np.ones(3), size=n_draws) + fulls, holds = [], [] + for W in draws: + nets = {} + for a, p in pre.items(): + d = direction_from_weights(p["S"], np.tile(W, (len(p["r"]), 1))) + tgt = np.clip(d * p["scal"], 0.0, 2.0) + tgt[~np.isfinite(tgt)] = 0.0 + pos = np.zeros(len(tgt)); pos[1:] = tgt[:-1] + turn = np.abs(np.diff(pos, prepend=0.0)) + net = pos * p["r"] - al.FEE_SIDE * turn + net[0] = 0.0 + nets[a] = pd.Series(net, index=p["idx"]) + s = combo_5050(nets) + f, h = sh_full_hold(s) + fulls.append(f); holds.append(h) + return np.array(fulls), np.array(holds), draws + + +# --------------------------------------------------------------------------- +def main(): + print("=" * 88) + print("r0702 REGIME-SPEED: pesi tra orizzonti TSMOM condizionati al regime di vol") + print("=" * 88) + + # ---- 1) SANITY: pesi fissi EQUAL devono riprodurre il baseline TP01 ------------ + ctrl = per_asset_series(fixed_target(EQ_W)) + ctrl_5050 = combo_5050(ctrl) + base = al.tp01_baseline_daily() + J = pd.concat({"mine": ctrl_5050, "tp01": base}, axis=1, join="inner").dropna() + mf, mh = sh_full_hold(J["mine"]); bf, bh = sh_full_hold(J["tp01"]) + max_diff = float(np.max(np.abs(J["mine"].values - J["tp01"].values))) + print(f"\n[SANITY] EQ-weight per-orizzonte vs TP01 canonico (50/50 daily):") + print(f" mine full {mf:+.3f} hold {mh:+.3f} tp01 full {bf:+.3f} hold {bh:+.3f}" + f" max|Δdaily-ret| = {max_diff:.2e}") + sanity_ok = abs(mf - bf) < 0.02 and max_diff < 1e-9 + print(f" sanity_ok = {sanity_ok}") + + # ---- 2) FAMIGLIA ONESTA: 16 celle, selezione in-sample + DSR automatici -------- + grid = [dict(thr=thr, mapping=m, blend=b, measure=meas) + for meas in ("rv", "dvol") + for thr in (0.60, 0.75) + for m in ("hv_fast", "hv_slow") + for b in ("hard", "linear")] + print(f"\n[FAMIGLIA] study_family_honest su {len(grid)} celle (1d)...") + fam = al.study_family_honest("R0702-REGIME-SPEED", factory, grid, tfs=("1d",)) + ch = fam["chosen"] + print(f" cella scelta IN-SAMPLE: {ch['params']} (IS Sharpe {ch['insample_sharpe']}," + f" full {ch['full_sharpe']})") + print(f" n_cells={fam['n_cells']} deflated_sharpe={fam['deflated_sharpe']}" + f" expected_null_max={fam['expected_null_max']} dsr_pass={fam['dsr_pass']}") + print(f" earns_slot_marginal={fam['earns_slot_marginal']} (atteso False: quasi-TP01)" + f" verdict marginale={fam['marginal']['marginal_verdict']}") + print(" tutte le celle (ordinate per IS Sharpe):") + for r in fam["rows"]: + print(f" IS {r['insample_sharpe']:+.3f} full {r['full_sharpe']:+.3f} {r['params']}") + + # ---- 3) DOMINANZA della cella scelta vs fixed-blend canonico ------------------- + chosen_fn = factory(**{"tf": ch["tf"], **ch["params"]}) + cand = per_asset_series(chosen_fn) + cand_5050 = combo_5050(cand) + dom = dominance_table(cand, ctrl) + print("\n[DOMINANZA] cella scelta vs fixed-blend canonico (Sharpe, netto 0.10% RT):") + for k, d in dom.items(): + print(f" {k:>4s}: FULL {d['cand_full']:+.3f} vs {d['ctrl_full']:+.3f} (Δ{d['d_full']:+.3f})" + f" HOLD {d['cand_hold']:+.3f} vs {d['ctrl_hold']:+.3f} (Δ{d['d_hold']:+.3f})") + dominates = all(d["d_full"] >= 0 and d["d_hold"] >= 0 for d in dom.values()) + print(f" DD 50/50: cand {dd_of(cand_5050)*100:.1f}% ctrl {dd_of(ctrl_5050)*100:.1f}%") + print(f" dominates_all_6 = {dominates}") + + mc = multicut(cand_5050, ctrl_5050) + mc_ok = all(v > 0 for v in mc.values()) + print(f" multi-cut ΔSharpe (50/50, dal taglio a fine): {mc} all_positive={mc_ok}") + + corr = float(pd.concat({"c": cand_5050, "b": ctrl_5050}, axis=1, join="inner") + .dropna().corr().iloc[0, 1]) + print(f" corr(cand, ctrl) daily = {corr:.4f} (attesa ~1: è un tilt di TP01)") + + # ---- 4) CAUSALITÀ --------------------------------------------------------------- + caus = al.causality_ok(chosen_fn, tf="1d") + print(f"\n[CAUSALITÀ] causality_ok = {caus['ok']} max_tail_diff={caus['max_tail_diff']}" + f" checked={caus['checked']}") + + # ---- 5) NULL: pesi fissi casuali ------------------------------------------------ + print(f"\n[NULL] {NULL_DRAWS} draw Dirichlet di pesi FISSI sui 3 orizzonti (50/50)...") + nf, nh, _ = null_fixed_weights() + cf, chd = sh_full_hold(cand_5050) + p_full = float(np.mean(nf <= cf)); p_hold = float(np.mean(nh <= chd)) + print(f" null FULL: mean {nf.mean():+.3f} p90 {np.percentile(nf, 90):+.3f}" + f" max {nf.max():+.3f} cella {cf:+.3f} → pctl {p_full:.3f}") + print(f" null HOLD: mean {nh.mean():+.3f} p90 {np.percentile(nh, 90):+.3f}" + f" max {nh.max():+.3f} cella {chd:+.3f} → pctl {p_hold:.3f}") + beats_null = p_full >= 0.90 and p_hold >= 0.90 + print(f" beats_null_p90 (FULL e HOLD) = {beats_null}") + + # ---- 6) RV vs DVOL come regime --------------------------------------------------- + print("\n[RV vs DVOL] migliore cella per misura (full / IS Sharpe):") + for meas in ("rv", "dvol"): + rows = [r for r in fam["rows"] if r["params"]["measure"] == meas] + if rows: + b = max(rows, key=lambda r: r["insample_sharpe"]) + print(f" {meas:>4s}: best-IS {b['insample_sharpe']:+.3f} (full {b['full_sharpe']:+.3f})" + f" {b['params']}") + + # ---- 7) VERDETTO ------------------------------------------------------------------ + crit = dict(sanity_ok=sanity_ok, dominates=dominates, multicut_ok=mc_ok, + dsr_pass=bool(fam["dsr_pass"]), beats_null_p90=beats_null, + causal=bool(caus["ok"])) + n_pass = sum(crit.values()) + if all(crit.values()): + verdict = "PASS" + elif crit["sanity_ok"] and crit["causal"] and crit["dominates"] and crit["multicut_ok"]: + verdict = "LEAD" + else: + verdict = "FAIL" + print(f"\n[VERDETTO] {verdict} criteri={crit} ({n_pass}/{len(crit)})") + return verdict + + +if __name__ == "__main__": + main() diff --git a/scripts/research/r0702_skeptic_offset.py b/scripts/research/r0702_skeptic_offset.py new file mode 100644 index 0000000..b830cc9 --- /dev/null +++ b/scripts/research/r0702_skeptic_offset.py @@ -0,0 +1,515 @@ +#!/usr/bin/env python +"""r0702_skeptic_offset.py — VERIFICA AVVERSARIALE INDIPENDENTE di r0702_tp01_offset.py. + +Linee d'attacco (tutte con codice INDIPENDENTE dal finding, cross-check contro le sue funzioni): + A. COSTRUZIONE: daily-offset ricostruito via floor-division su epoca ms (niente pandas.resample); + h=0 deve == al.get('1d') e == tp01_baseline_daily; mapping daily->1h via searchsorted (niente + merge_asof); guardia troncamento del feed 1h (nessun look-ahead a h!=0). + B. STATISTICA: block-bootstrap congiunto delle 24 ancore sull'hold-out — lo spike di h=0 + (Sh(h0) - mediana(altri)) e' speciale o e' il massimo atteso di 24 stime correlate? + + hold-out finti (2020..2024): l'ancora migliore e' stabile o gira a caso? + C. TRANCHING: identita' K=4 == EW dei 4 book ancorati (netting non nasconde nulla)? + turnover verificato; DD del K=4 vs DD della ROTAZIONE TIPICA (non vs h=0 sfortunato); + bootstrap appaiato della differenza IS. + D. IMPATTO: blend TP+SKH 75/25 e book 5-sleeve ricalcolati con TP01 alle 24 ancore. + +Nessun file toccato fuori da questo script. Runtime ~3-6 min (SKH/XS/VRP/GTAA inclusi). +""" +from __future__ import annotations + +import sys +from functools import lru_cache +from pathlib import Path + +import numpy as np +import pandas as pd + +ROOT = Path("/opt/docker/PythagorasGoal") +sys.path.insert(0, str(ROOT / "scripts" / "research" / "alt")) +sys.path.insert(0, str(ROOT / "scripts" / "research")) +sys.path.insert(0, str(ROOT)) + +import altlib as al # noqa: E402 +import r0702_tp01_offset as RF # il finding, SOLO per cross-check # noqa: E402 +from src.strategies.trend_portfolio import CANONICAL, TrendPortfolio # noqa: E402 + +TP = TrendPortfolio(**CANONICAL) +HOLDOUT = al.HOLDOUT +ASSETS = ("BTC", "ETH") +MS_H = 3_600_000 +MS_D = 86_400_000 +FEE = al.FEE_SIDE +RNG = np.random.default_rng(42) +B_BOOT = 4000 +BLOCK = 20 + + +# =========================================================================== +# A. COSTRUZIONE INDIPENDENTE +# =========================================================================== +@lru_cache(maxsize=8) +def get1h(asset: str) -> pd.DataFrame: + return al.get(asset, "1h") + + +@lru_cache(maxsize=64) +def sk_daily(asset: str, h: int) -> pd.DataFrame: + """Daily-offset costruito a mano: day_id = (ts - h*1h) // 24h su epoca ms (open-labeled).""" + df = get1h(asset) + ts = df["timestamp"].values.astype(np.int64) + day = (ts - h * MS_H) // MS_D + uday, first = np.unique(day, return_index=True) + o = df["open"].values.astype(float) + hi = df["high"].values.astype(float) + lo = df["low"].values.astype(float) + c = df["close"].values.astype(float) + v = df["volume"].values.astype(float) + last = np.r_[first[1:], len(ts)] - 1 + out = pd.DataFrame(dict( + timestamp=uday * MS_D + h * MS_H, + open=o[first], + high=np.maximum.reduceat(hi, first), + low=np.minimum.reduceat(lo, first), + close=c[last], + volume=np.add.reduceat(v, first), + )) + out["datetime"] = pd.to_datetime(out["timestamp"], unit="ms", utc=True) + return out + + +def sk_net_daily(asset: str, h: int) -> pd.Series: + """Rendimenti netti TP01 sul grid daily-offset (pipeline mia: shift+fee espliciti).""" + d = sk_daily(asset, h) + c = d["close"].values.astype(float) + r = np.zeros(len(c)); r[1:] = c[1:] / c[:-1] - 1.0 + tgt = TP.target_series(d) + pos = np.zeros(len(tgt)); pos[1:] = tgt[:-1] + net = pos * r - FEE * np.abs(np.diff(pos, prepend=0.0)) + net[0] = 0.0 + return pd.Series(net, index=pd.DatetimeIndex(d["datetime"])) + + +@lru_cache(maxsize=32) +def sk_port_daily(h: int) -> pd.Series: + J = pd.concat({a: sk_net_daily(a, h) for a in ASSETS}, axis=1, join="inner").fillna(0.0) + return al._to_daily(0.5 * J["BTC"] + 0.5 * J["ETH"]) + + +def sk_pos_hourly(asset: str, hs: tuple, df1h: pd.DataFrame | None = None) -> np.ndarray: + """Posizione TENUTA durante ogni barra 1h (ensemble media delle ancore hs), via searchsorted: + pos durante barra i = target dell'ultima barra daily-offset con close nominale <= open(barra i).""" + df = get1h(asset) if df1h is None else df1h + open_ms = df["timestamp"].values.astype(np.int64) + pos = np.zeros(len(open_ms)) + for h in hs: + d = sk_daily(asset, h) if df1h is None else sk_daily_from(df, h) + tgt = np.nan_to_num(TP.target_series(d), nan=0.0) + close_ms = d["timestamp"].values.astype(np.int64) + MS_D + j = np.searchsorted(close_ms, open_ms, side="right") - 1 + p = np.where(j >= 0, tgt[np.clip(j, 0, None)], 0.0) + pos += p / len(hs) + return pos + + +def sk_daily_from(df1h: pd.DataFrame, h: int) -> pd.DataFrame: + """sk_daily ma da un frame 1h arbitrario (per il test di troncamento).""" + ts = df1h["timestamp"].values.astype(np.int64) + day = (ts - h * MS_H) // MS_D + uday, first = np.unique(day, return_index=True) + c = df1h["close"].values.astype(float) + last = np.r_[first[1:], len(ts)] - 1 + out = pd.DataFrame(dict(timestamp=uday * MS_D + h * MS_H, close=c[last])) + out["datetime"] = pd.to_datetime(out["timestamp"], unit="ms", utc=True) + return out + + +def sk_book_hourly(hs: tuple) -> tuple[pd.Series, float, dict]: + """Book 0.5/0.5 sul grid 1h con posizioni ensemble; ritorna (daily, turnover/y, per-asset net).""" + nets, turns = {}, 0.0 + for a in ASSETS: + df = get1h(a) + c = df["close"].values.astype(float) + r = np.zeros(len(c)); r[1:] = c[1:] / c[:-1] - 1.0 + pos = sk_pos_hourly(a, hs) + turn = np.abs(np.diff(pos, prepend=0.0)) + net = pos * r - FEE * turn + net[0] = 0.0 + idx = pd.DatetimeIndex(pd.to_datetime(df["datetime"], utc=True)) + nets[a] = pd.Series(net, index=idx) + yrs = len(net) / (24 * 365.25) + turns += 0.5 * turn.sum() / yrs + J = pd.concat(nets, axis=1, join="inner").fillna(0.0) + return al._to_daily(0.5 * J["BTC"] + 0.5 * J["ETH"]), turns, nets + + +def sh3(s: pd.Series) -> tuple[float, float, float]: + return (al._sh(s), al._sh(s[s.index < HOLDOUT]), al._sh(s[s.index >= HOLDOUT])) + + +def part_A() -> None: + print("=" * 100) + print("A. COSTRUZIONE — ricostruzione indipendente (floor-division ms / searchsorted)") + print("=" * 100) + # A1: daily-offset mio vs al.get('1d') (h=0) e vs il loro resample_offset (h campionati) + for a in ASSETS: + ref = al.get(a, "1d") + mine0 = sk_daily(a, 0) + assert len(mine0) == len(ref), f"A1 len mismatch {a}" + for col in ("timestamp", "open", "high", "low", "close", "volume"): + rtol = 1e-9 if col == "volume" else 0.0 # volume: solo ordine di sommatoria float + assert np.allclose(mine0[col].values.astype(float), ref[col].values.astype(float), + atol=0, rtol=rtol), f"A1 h=0 mismatch {a}:{col}" + for h in (1, 5, 11, 13, 21, 23): + theirs = RF.daily_off(a, h) + m = sk_daily(a, h) + assert len(m) == len(theirs), f"A1 len mismatch {a} h={h}" + for col in ("timestamp", "open", "high", "low", "close", "volume"): + rtol = 1e-9 if col == "volume" else 0.0 + assert np.allclose(m[col].values.astype(float), + theirs[col].values.astype(float), atol=0, rtol=rtol), \ + f"A1 h={h} mismatch {a}:{col}" + print("[A1] daily-offset: costruzione mia == al.get('1d') (h=0) == loro resample_offset " + "(h=1,5,11,13,21,23, tutte le colonne, bit-exact): OK") + + # A2: pipeline completa h=0 vs baseline del progetto + mine = sk_port_daily(0) + base = al.tp01_baseline_daily() + assert len(mine) == len(base) and np.allclose(mine.values, base.values, atol=1e-12), "A2 FAIL" + f, i, ho = sh3(mine) + print(f"[A2] portafoglio h=0 (pipeline mia) == tp01_baseline_daily: OK " + f"(FULL {f:.4f} / IS {i:.4f} / HOLD {ho:.4f})") + + # A3: troncamento del feed 1h -> posizioni orarie IDENTICHE su tutto il range troncato + for a in ASSETS: + df = get1h(a) + for cut in (len(df) - 3000, len(df) - 777): + dtr = df.iloc[:cut].reset_index(drop=True) + for h in (0, 5, 13, 21): + p_full = sk_pos_hourly(a, (h,)) + p_tr = sk_pos_hourly(a, (h,), df1h=dtr) + assert np.allclose(p_full[:cut], p_tr, atol=1e-12), \ + f"A3 look-ahead {a} h={h} cut={cut}" + print("[A3] troncamento 1h (2 cut x 4 ancore x 2 asset): posizioni orarie invariate " + "sul prefisso -> nessun look-ahead nel mapping daily->1h: OK") + + # A4: vol-target ricalcolata per-offset? (fatto strutturale + evidenza numerica) + for a in ASSETS: + for h in (5, 13): + assert TP._bpd(sk_daily(a, h)) == 1, "A4 bpd" + t0 = TP.target_series(sk_daily(a, 0)) + t13 = TP.target_series(sk_daily(a, 13)) + m = min(len(t0), len(t13)) + d = np.abs(t0[300:m] - t13[300:m]) + print(f"[A4] {a}: target h=0 vs h=13 stesso giorno-calendario, |diff| media " + f"{np.nanmean(d):.4f} (max {np.nanmax(d):.3f}) -> vol e segnale RICALCOLATI " + f"sul grid dell'ancora (target_series riceve il grid offset)") + + # A5: cross-check book orario mio vs loro (K=1 h0 e K=4) + for name, hs in (("K=1 h0", (0,)), ("K=4", (0, 6, 12, 18))): + mine_s, mine_t, _ = sk_book_hourly(hs) + theirs_s, theirs_t = RF.port_hourly(hs) + common = mine_s.index.intersection(theirs_s.index) + dmax = float(np.max(np.abs(mine_s.loc[common].values - theirs_s.loc[common].values))) + print(f"[A5] {name}: book 1h mio vs loro — max|diff ret giornaliero| {dmax:.2e}, " + f"turn/y {mine_t:.2f} vs {theirs_t:.2f}") + assert dmax < 1e-10, f"A5 mismatch {name}" + + +# =========================================================================== +# B. STATISTICA — lo spike h=0 e' speciale? +# =========================================================================== +@lru_cache(maxsize=2) +def anchor_matrix() -> pd.DataFrame: + cols = {f"h{h:02d}": sk_port_daily(h) for h in range(24)} + return pd.concat(cols, axis=1, join="inner").dropna() + + +def _sh_mat(R: np.ndarray) -> np.ndarray: + mu = R.mean(axis=1) + sd = R.std(axis=1) + return np.where(sd > 0, mu / sd, 0.0) * np.sqrt(365.25) + + +def block_boot_stats(M: np.ndarray, B: int, block: int, rng) -> dict: + n, K = M.shape + nblocks = int(np.ceil(n / block)) + g0s, gmaxs, med_all, sh0s = [], [], [], [] + done = 0 + while done < B: + b = min(500, B - done) + starts = rng.integers(0, n, size=(b, nblocks)) + idx = (starts[:, :, None] + np.arange(block)[None, None, :]) % n + idx = idx.reshape(b, -1)[:, :n] + R = M[idx] # (b, n, K) + Sh = np.stack([_sh_mat(R[:, :, k]) for k in range(K)], axis=1) + med_others = np.empty_like(Sh) + for h in range(K): + others = np.delete(Sh, h, axis=1) + med_others[:, h] = np.median(others, axis=1) + g = Sh - med_others + g0s.append(g[:, 0]) + gmaxs.append(g.max(axis=1)) + med_all.append(np.median(Sh, axis=1)) + sh0s.append(Sh[:, 0]) + done += b + return dict(g0=np.concatenate(g0s), gmax=np.concatenate(gmaxs), + med=np.concatenate(med_all), sh0=np.concatenate(sh0s)) + + +def part_B() -> None: + print("\n" + "=" * 100) + print("B. STATISTICA — spike h=0 sull'hold-out: speciale o massimo atteso di 24 stime correlate?") + print("=" * 100) + Mdf = anchor_matrix() + Mh = Mdf[Mdf.index >= HOLDOUT].values + sh_hold = _sh_mat(Mh.T) + med_others_obs = np.median(sh_hold[1:]) + g0_obs = sh_hold[0] - med_others_obs + corr = np.corrcoef(Mh.T) + iu = np.triu_indices(24, 1) + print(f"hold-out: {Mh.shape[0]} giorni, 24 ancore; Sh h=0 {sh_hold[0]:.3f}, " + f"mediana altri {med_others_obs:.3f}, spike osservato g0 = {g0_obs:.3f}") + print(f"correlazione daily fra ancore (hold-out): mediana {np.median(corr[iu]):.3f}, " + f"min {corr[iu].min():.3f}") + + for blk in (10, 20, 40): + bs = block_boot_stats(Mh, B_BOOT, blk, np.random.default_rng(42 + blk)) + p_any = float(np.mean(bs["gmax"] >= g0_obs)) + p_g0 = float(np.mean(bs["g0"] <= 0.0)) + ci_g0 = np.percentile(bs["g0"], [2.5, 97.5]) + ci_med = np.percentile(bs["med"], [2.5, 97.5]) + ci_sh0 = np.percentile(bs["sh0"], [2.5, 97.5]) + print(f" block={blk:>2}: P(max-spike di UNA QUALSIASI ancora >= {g0_obs:.2f}) = " + f"{p_any:.3f} | P(g0<=0) = {p_g0:.3f} | CI95 g0 [{ci_g0[0]:+.2f},{ci_g0[1]:+.2f}] " + f"| CI95 Sh mediana-ancore [{ci_med[0]:+.2f},{ci_med[1]:+.2f}] " + f"| CI95 Sh h=0 [{ci_sh0[0]:+.2f},{ci_sh0[1]:+.2f}]") + + # hold-out finti: l'ancora migliore per finestra e' stabile? + print("\n finestre annuali (hold-out finti) — best/worst anchor, h=0, spread:") + print(f" {'finestra':<9} {'best':>5} {'ShBest':>7} {'worst':>6} {'ShWorst':>8} " + f"{'mediana':>8} {'h=0':>6} {'pctl h0':>8} {'max-med':>8}") + years = [2020, 2021, 2022, 2023, 2024] + windows: list[tuple[str, pd.DataFrame]] = [ + (str(y), Mdf[Mdf.index.year == y]) for y in years] + [("2025+", Mdf[Mdf.index >= HOLDOUT])] + sh_by_win = {} + from scipy.stats import spearmanr + for name, W in windows: + sh = _sh_mat(W.values.T) + sh_by_win[name] = sh + pctl0 = float((sh < sh[0]).mean() + 0.5 * (sh == sh[0]).mean()) * 100 + print(f" {name:<9} {int(np.argmax(sh)):>5} {sh.max():>7.3f} {int(np.argmin(sh)):>6} " + f"{sh.min():>8.3f} {np.median(sh):>8.3f} {sh[0]:>6.3f} {pctl0:>7.0f}° " + f"{sh.max() - np.median(sh):>8.3f}") + names = [n for n, _ in windows] + print("\n stabilita' del ranking ancore (Spearman fra finestre consecutive):") + for a, b in zip(names[:-1], names[1:]): + rho, p = spearmanr(sh_by_win[a], sh_by_win[b]) + print(f" {a} vs {b}: rho={rho:+.2f} (p={p:.2f})") + # l'ancora migliore di ogni finestra, quanto rende NELLE ALTRE finestre? (pctl medio) + print(" best-anchor di ogni finestra valutata nelle ALTRE finestre (pctl medio su 24):") + for name in names: + h_star = int(np.argmax(sh_by_win[name])) + pct = [float((sh_by_win[o] < sh_by_win[o][h_star]).mean()) * 100 + for o in names if o != name] + print(f" best({name}) = h={h_star:>2} -> pctl medio altrove {np.mean(pct):.0f}° " + f"(per finestra: {', '.join(f'{p:.0f}' for p in pct)})") + + # ritorno totale hold-out per ancora (per la narrativa '+3.5%') + tot = np.prod(1 + Mh, axis=0) - 1 + print(f"\n ritorno TOTALE hold-out per ancora: min {tot.min():+.1%} / mediana " + f"{np.median(tot):+.1%} / max {tot.max():+.1%} (h=0: {tot[0]:+.1%})") + dd = [al._dd_ret(pd.Series(Mh[:, k])) for k in range(24)] + print(f" maxDD hold-out per ancora: min {min(dd):.1%} / mediana {np.median(dd):.1%} / " + f"max {max(dd):.1%} (h=0: {dd[0]:.1%}) [B&H 50/50 2025-26: DD ~60%]") + + +# =========================================================================== +# C. TRANCHING — gratis davvero? +# =========================================================================== +def part_C() -> None: + print("\n" + "=" * 100) + print("C. TRANCHING — identita' EW, turnover, DD vs rotazione tipica, significativita' IS") + print("=" * 100) + + # C1: K=4 book == EW dei 4 book ancorati? (identita' esatta, incluse fee) + for a in ASSETS: + df = get1h(a) + c = df["close"].values.astype(float) + r = np.zeros(len(c)); r[1:] = c[1:] / c[:-1] - 1.0 + hs = (0, 6, 12, 18) + pos_e = sk_pos_hourly(a, hs) + net_e = pos_e * r - FEE * np.abs(np.diff(pos_e, prepend=0.0)); net_e[0] = 0.0 + nets_1 = [] + turns_1 = [] + for h in hs: + p = sk_pos_hourly(a, (h,)) + t = np.abs(np.diff(p, prepend=0.0)) + n1 = p * r - FEE * t; n1[0] = 0.0 + nets_1.append(n1) + turns_1.append(t.sum()) + ew = np.mean(nets_1, axis=0) + turn_e = np.abs(np.diff(pos_e, prepend=0.0)).sum() + print(f"[C1] {a}: max|net K4 - EW(4 book singoli)| = {np.max(np.abs(net_e - ew)):.2e} ; " + f"turnover K4 {turn_e:.1f} vs media singoli {np.mean(turns_1):.1f} " + f"(rapporto {turn_e / np.mean(turns_1):.4f})") + + # C2: tutte le rotazioni (mie): livelli e dispersione, DD compreso + fams = {"singole(24)": [(h,) for h in range(24)], + "K=2(12)": [(h, h + 12) for h in range(12)], + "K=4(6)": [tuple(h + 6 * j for j in range(4)) for h in range(6)]} + stats = {} + for fam, rots in fams.items(): + rec = [] + for hs in rots: + s, t, _ = sk_book_hourly(hs) + f, i, ho = sh3(s) + rec.append(dict(hs=hs, full=f, is_=i, hold=ho, dd=al._dd_ret(s), + dd_h=al._dd_ret(s[s.index >= HOLDOUT]), turn=t)) + stats[fam] = pd.DataFrame(rec) + print("\n[C2] rotazioni complete (book 1h, misura identica per tutte):") + print(f" {'famiglia':<12} {'IS med[min,max]':>24} {'HOLD med[min,max]':>26} " + f"{'maxDD med[min,max]':>24} {'turn/y med':>10}") + for fam, T in stats.items(): + print(f" {fam:<12} {T.is_.median():>8.3f} [{T.is_.min():.3f},{T.is_.max():.3f}]" + f" {T.hold.median():>9.3f} [{T.hold.min():+.3f},{T.hold.max():+.3f}]" + f" {T.dd.median():>8.1%} [{T.dd.min():.1%},{T.dd.max():.1%}]" + f" {T.turn.median():>8.2f}") + s24, _, _ = sk_book_hourly(tuple(range(24))) + f24, i24, h24 = sh3(s24) + print(f" K=24 IS {i24:.3f} HOLD {h24:+.3f} maxDD {al._dd_ret(s24):.1%}") + T1 = stats["singole(24)"] + T4 = stats["K=4(6)"] + print(f"\n -> claim 'maxDD 14.7->11.9': h=0 singolo DD {T1.dd.iloc[0]:.1%} ma la MEDIANA " + f"delle 24 singole e' {T1.dd.median():.1%}; K=4 mediano {T4.dd.median():.1%} " + f"=> beneficio del tranching vs ancora TIPICA = {T1.dd.median() - T4.dd.median():+.1%}pt, " + f"vs h=0 = {T1.dd.iloc[0] - T4.dd.median():+.1%}pt (in gran parte 'h=0 era sfortunato sul DD')") + print(f" -> claim 'IS 1.49->1.54/1.56': mediana IS delle 24 singole = {T1.is_.median():.3f} " + f"(K=4 mediano {T4.is_.median():.3f}) => il 'miglioramento' e' tornare alla MEDIA delle " + f"ancore, h=0 era al {(T1.is_ < T1.is_.iloc[0]).mean() * 100:.0f}° pctl IS") + + # C3: significativita' IS del K=4 vs h=0 (bootstrap appaiato a blocchi) + s0 = sk_book_hourly((0,))[0] + s4 = sk_book_hourly((0, 6, 12, 18))[0] + common = s0.index.intersection(s4.index) + A = s4.loc[common]; Bser = s0.loc[common] + mask = common < HOLDOUT + Ai, Bi = A[mask].values, Bser[mask].values + n = len(Ai) + nblocks = int(np.ceil(n / BLOCK)) + d_obs = al._sh(A[mask]) - al._sh(Bser[mask]) + ds = [] + rng = np.random.default_rng(7) + for _ in range(B_BOOT // 500): + starts = rng.integers(0, n, size=(500, nblocks)) + idx = (starts[:, :, None] + np.arange(BLOCK)[None, None, :]) % n + idx = idx.reshape(500, -1)[:, :n] + Ra, Rb = Ai[idx], Bi[idx] + sa = Ra.mean(1) / Ra.std(1) * np.sqrt(365.25) + sb = Rb.mean(1) / Rb.std(1) * np.sqrt(365.25) + ds.append(sa - sb) + ds = np.concatenate(ds) + print(f"\n[C3] IS: Sh(K4) - Sh(h0) = {d_obs:+.3f}; bootstrap appaiato (block {BLOCK}, " + f"B={len(ds)}): CI95 [{np.percentile(ds, 2.5):+.3f}, {np.percentile(ds, 97.5):+.3f}], " + f"P(diff<=0) = {np.mean(ds <= 0):.3f}") + # e vs l'ancora mediana (piu' onesto): K4 confrontato con OGNI singola + dvs = [d for h in range(24) + for d in [al._sh(A[mask]) - al._sh(sk_book_hourly((h,))[0].loc[common][mask])]] + print(f" Sh_IS(K4) - Sh_IS(singola h) sulle 24 ancore: min {min(dvs):+.3f} / " + f"mediana {np.median(dvs):+.3f} / max {max(dvs):+.3f} " + f"-> vs ancora tipica il guadagno IS e' ~{np.median(dvs):+.2f}, non +0.05/+0.07") + + # C4: small-cap $600 (mia implementazione min-order) + print("\n[C4] small-cap $600 (min order $5, quota 0.5/asset):") + for name, hs in (("K=1 h0", (0,)), ("K=2", (0, 12)), ("K=4", (0, 6, 12, 18)), + ("K=24", tuple(range(24)))): + nets_r, nets_m, ntr = {}, {}, 0 + for a in ASSETS: + df = get1h(a) + c = df["close"].values.astype(float) + r = np.zeros(len(c)); r[1:] = c[1:] / c[:-1] - 1.0 + t = 0.5 * sk_pos_hourly(a, hs) + held = np.empty(len(t)); cur = 0.0 + for i in range(len(t)): + if abs(t[i] - cur) * 600.0 >= 5.0: + cur = t[i]; ntr += 1 + held[i] = cur + pos = np.zeros(len(held)); pos[1:] = held[:-1] + nr = pos * r - FEE * np.abs(np.diff(pos, prepend=0.0)); nr[0] = 0.0 + posm = np.zeros(len(t)); posm[1:] = t[:-1] + nm = posm * r - FEE * np.abs(np.diff(posm, prepend=0.0)); nm[0] = 0.0 + idx = pd.DatetimeIndex(pd.to_datetime(df["datetime"], utc=True)) + nets_r[a] = pd.Series(nr, index=idx); nets_m[a] = pd.Series(nm, index=idx) + Jr = pd.concat(nets_r, axis=1, join="inner").fillna(0.0) + Jm = pd.concat(nets_m, axis=1, join="inner").fillna(0.0) + dr = al._to_daily(Jr["BTC"] + Jr["ETH"]); dm = al._to_daily(Jm["BTC"] + Jm["ETH"]) + yrs = len(dr) / 365.25 + print(f" {name:<8} Sh real {al._sh(dr):.3f} (model {al._sh(dm):.3f}, haircut " + f"{al._sh(dm) - al._sh(dr):+.3f}) trade/y {ntr / yrs:.0f}") + + +# =========================================================================== +# D. IMPATTO sui numeri del progetto (blend SKH e book 5-sleeve, TP01 per ancora) +# =========================================================================== +def part_D() -> None: + print("\n" + "=" * 100) + print("D. IMPATTO — blend TP+SKH 75/25 e book 5-sleeve con TP01 alle 24 ancore") + print("=" * 100) + from src.portfolio.portfolio import combine_outer, to_daily + try: + from src.portfolio.sleeves import (_gtaa_daily_returns, _skyhook_returns, + _vrp_combo_returns, _xsec_returns) + skh = to_daily(_skyhook_returns()) + except Exception as e: + print(f" [SKIP] sleeve non calcolabili: {type(e).__name__}: {e}") + return + + def hold_sh(s: pd.Series) -> float: + return al._sh(s[s.index >= HOLDOUT]) + + # blend deribit book 75/25 + blends = [] + for h in range(24): + tp = sk_port_daily(h) + b = combine_outer({"TP": tp, "SKH": skh}, {"TP": 0.75, "SKH": 0.25}) + b = b[b.index >= tp.index.min()] + blends.append(hold_sh(b)) + b24 = combine_outer({"TP": sk_book_hourly(tuple(range(24)))[0], "SKH": skh}, + {"TP": 0.75, "SKH": 0.25}) + print(f"[D1] blend 0.75*TP01(h)+0.25*SKH — Sharpe HOLD: h=0 {blends[0]:.2f} | " + f"min {min(blends):.2f} / mediana {np.median(blends):.2f} / max {max(blends):.2f} | " + f"TP=K24 {hold_sh(b24[b24.index >= sk_port_daily(0).index.min()]):.2f} " + f"(claim del progetto: 0.31 -> 1.17)") + + # book 5-sleeve (pesi CLAUDE.md), attivazione era crypto + try: + cols_fixed = dict(XS=to_daily(_xsec_returns()), VRP=to_daily(_vrp_combo_returns()), + SKH=skh, GTAA=to_daily(_gtaa_daily_returns())) + except Exception as e: + print(f" [SKIP 5-sleeve] {type(e).__name__}: {e}") + return + W = dict(TP=0.33, XS=0.15, VRP=0.12, SKH=0.20, GTAA=0.20) + lo = sk_port_daily(0).index.min() + books = [] + for h in range(24): + cols = dict(TP=sk_port_daily(h), **cols_fixed) + s = combine_outer(cols, W, lo=lo) + books.append((hold_sh(s), al._sh(s))) + bh = [b[0] for b in books]; bf = [b[1] for b in books] + s24b = combine_outer(dict(TP=sk_book_hourly(tuple(range(24)))[0], **cols_fixed), W, lo=lo) + print(f"[D2] book 5-sleeve (TP 33/XS 15/VRP 12/SKH 20/GTAA 20) — Sharpe HOLD: " + f"h=0 {bh[0]:.2f} | min {min(bh):.2f} / mediana {np.median(bh):.2f} / max {max(bh):.2f} " + f"| TP=K24 {hold_sh(s24b):.2f}") + print(f" Sharpe FULL: h=0 {bf[0]:.2f} | min {min(bf):.2f} / mediana {np.median(bf):.2f} " + f"/ max {max(bf):.2f} | TP=K24 {al._sh(s24b):.2f}") + + +def main() -> None: + part_A() + part_B() + part_C() + part_D() + print("\nFatto (scettico).") + + +if __name__ == "__main__": + main() diff --git a/scripts/research/r0702_slow_clock.py b/scripts/research/r0702_slow_clock.py new file mode 100644 index 0000000..676e872 --- /dev/null +++ b/scripts/research/r0702_slow_clock.py @@ -0,0 +1,319 @@ +"""r0702_slow_clock.py — FILONE: clock più lenti del daily + banded rebalancing per TP01. + +Due idee di TIMING DI ESECUZIONE (non di segnale) sul TP01 CANONICAL (PORT LF1d): + (A) CLOCK LENTI — segnale calcolato daily, posizione aggiornata solo ogni N giorni + (N in {2,3,5,7}). ⚠ timing luck: si riportano TUTTE le N fasi (min/med/max) e + l'ENSEMBLE delle fasi (media dei libri sfasati), MAI la fase migliore da sola. + (B) BANDE DI ISTERESI — decisione daily, si esegue solo se |target − posizione| > banda + (banda in frazione di equity PER ASSET, in {0, .025, .05, .10, .20}); quando si + esegue si va al target pieno. + +Onestà: + - selezione cella SOLO in-sample pre-2025 (pattern al.select_cell_insample); l'hold-out + della cella scelta si RIPORTA, non si sceglie. + - deflated Sharpe (al.deflated_sharpe) su TUTTI i trial esplorati (fasi incluse). + - Sharpe LORDO (fee=0) accanto al netto: una variante di esecuzione onesta ha lordo + ~uguale al canonico e netto >= (il guadagno è meccanico-di-costo, non fitting). + - executability: replica di eval_weights_smallcap a capitale 600/2000/10000 (min order + $5, capitale per-asset = C/2) per baseline vs variante scelta — a $600 la banda + implicita min-order è 5/(600/2) ≈ 0.0167 di peso per asset. + - causalità: target TP01 causale (verificato altrove); i filtri di esecuzione usano solo + stato passato; eval_weights fa lo shift +1; check prefix-consistency inline sulla + cella scelta. Nessun ffill mixed-TF, nessun .view("int64") su tz-aware. + +Convenzione (stessa di eval_weights/TrendPortfolio): il peso resta costante tra i +ribilanciamenti (fee solo su |Δpeso|); il drift del peso intra-periodo non è modellato +(secondo ordine a N<=7 giorni) — dichiarato nei caveat. + +Run: uv run python scripts/research/r0702_slow_clock.py +""" +from __future__ import annotations + +import sys + +sys.path.insert(0, "/opt/docker/PythagorasGoal/scripts/research/alt") +import altlib as al # noqa: E402 + +import numpy as np # noqa: E402 +import pandas as pd # noqa: E402 + +from src.strategies.trend_portfolio import CANONICAL, TrendPortfolio # noqa: E402 + +HOLDOUT = al.HOLDOUT +FEE = al.FEE_SIDE +EPOCH = pd.Timestamp("1970-01-01", tz="UTC") + +CLOCK_NS = (2, 3, 5, 7) +BANDS = (0.0, 0.025, 0.05, 0.10, 0.20) +CAPITALS = (600.0, 2000.0, 10000.0) +MIN_ORDER = 5.0 + + +# --------------------------------------------------------------------------- +# targets & execution filters (tutti causali: stato = solo passato) +# --------------------------------------------------------------------------- +def daily_targets() -> dict[str, tuple[pd.DataFrame, np.ndarray]]: + tp = TrendPortfolio(**CANONICAL) + out = {} + for a in al.CERTIFIED: + df = al.get(a, "1d") + out[a] = (df, tp.target_series(df)) + return out + + +def epoch_days(df: pd.DataFrame) -> np.ndarray: + dt = pd.to_datetime(df["datetime"], utc=True) + return ((dt.dt.floor("D") - EPOCH) // pd.Timedelta(days=1)).values.astype(int) + + +def slow_clock_exec(df: pd.DataFrame, tgt: np.ndarray, N: int, phase: int) -> np.ndarray: + """Aggiorna la posizione solo nei giorni con epoch_day % N == phase (ancoraggio a + calendario -> prefix-consistent, entrambe le gambe ribilanciano lo stesso giorno).""" + days = epoch_days(df) + out = np.empty(len(tgt)) + cur = 0.0 + for i in range(len(tgt)): + if days[i] % N == phase: + cur = tgt[i] + out[i] = cur + return out + + +def band_exec(tgt: np.ndarray, band: float) -> np.ndarray: + """Esegue (va al target pieno) solo se |target − posizione corrente| > band.""" + out = np.empty(len(tgt)) + cur = 0.0 + for i in range(len(tgt)): + if abs(tgt[i] - cur) > band: + cur = tgt[i] + out[i] = cur + return out + + +def smallcap_exec(tgt: np.ndarray, capital_per_asset: float, + min_order: float = MIN_ORDER) -> np.ndarray: + """Replica della logica di al.eval_weights_smallcap (serve la SERIE, non solo le + metriche): un Δpeso il cui nozionale < min_order NON si esegue.""" + out = np.empty(len(tgt)) + cur = 0.0 + for i in range(len(tgt)): + if abs(tgt[i] - cur) * capital_per_asset >= min_order: + cur = tgt[i] + out[i] = cur + return out + + +# --------------------------------------------------------------------------- +# valutazione book 50/50 (netto + lordo) +# --------------------------------------------------------------------------- +def _series(df: pd.DataFrame, et: np.ndarray, fee_side: float) -> pd.Series: + ev = al.eval_weights(df, et, fee_side=fee_side) + return pd.Series(ev["net"], index=ev["idx"]) + + +def book_eval(pairs: dict[str, tuple[pd.DataFrame, np.ndarray]]) -> dict: + """pairs: {asset: (df, exec_target)} -> metriche book 50/50 nette e lorde.""" + net_s, gro_s = {}, {} + turn_book = 0.0 + orders_y = 0.0 + for a, (df, et) in pairs.items(): + net_s[a] = _series(df, et, FEE) + gro_s[a] = _series(df, et, 0.0) + years = (df["datetime"].iloc[-1] - df["datetime"].iloc[0]).total_seconds() / 86400 / 365.25 + pos = np.zeros(len(et)); pos[1:] = et[:-1] + turn = np.abs(np.diff(pos, prepend=0.0)) + turn_book += 0.5 * turn.sum() / years # in unità di equity del book + orders_y += float(np.sum(turn > 1e-12) / years) # ordini reali (entrambe le gambe) + NET = pd.concat(net_s, axis=1, join="inner").fillna(0.0) + GRO = pd.concat(gro_s, axis=1, join="inner").fillna(0.0) + assets = list(pairs) + net = 0.5 * NET[assets[0]] + 0.5 * NET[assets[1]] + gro = 0.5 * GRO[assets[0]] + 0.5 * GRO[assets[1]] + + def _hold(s): return s[s.index >= HOLDOUT] + def _ins(s): return s[s.index < HOLDOUT] + + eqn = np.cumprod(1.0 + np.clip(net.values, -0.99, None)) + span_y = (net.index[-1] - net.index[0]).total_seconds() / 86400 / 365.25 + cagr = eqn[-1] ** (1 / span_y) - 1 if span_y > 0 else 0.0 + return dict( + sh_full_net=al._sh(net), sh_hold_net=al._sh(_hold(net)), sh_ins_net=al._sh(_ins(net)), + sh_full_gro=al._sh(gro), sh_hold_gro=al._sh(_hold(gro)), + maxdd=al._dd_ret(net), cagr=cagr, + turnover_y=turn_book, orders_y=orders_y, net=net, gross=gro, + ) + + +def row(tag: str, m: dict) -> str: + return (f"{tag:<26} | net F {m['sh_full_net']:5.2f} H {m['sh_hold_net']:5.2f} " + f"(IS {m['sh_ins_net']:5.2f}) | gross F {m['sh_full_gro']:5.2f} " + f"H {m['sh_hold_gro']:5.2f} | DD {m['maxdd']*100:4.1f}% | CAGR {m['cagr']*100:5.1f}% " + f"| turn/y {m['turnover_y']:5.1f} | ord/y {m['orders_y']:6.1f}") + + +# --------------------------------------------------------------------------- +def main() -> None: + pairs = daily_targets() + + # ---- sanity check: riproduci al.tp01_baseline_daily() ------------------- + base = book_eval(pairs) + ref = al.tp01_baseline_daily() + common = base["net"].index.intersection(ref.index) + diff = float(np.max(np.abs(base["net"].reindex(common).values - ref.reindex(common).values))) + print("=" * 118) + print("SANITY — baseline daily vs al.tp01_baseline_daily():", + f"max|Δdaily ret| = {diff:.2e}", + f"(Sharpe qui {base['sh_full_net']:.3f} / ref {al._sh(ref):.3f})") + assert diff < 1e-9, "baseline non riprodotta!" + print(row("BASELINE daily band=0", base)) + + all_trial_sharpes: list[float] = [base["sh_full_net"]] + candidates: dict[str, dict] = {"baseline_daily": base} + + # ---- (A) CLOCK LENTI: tutte le fasi + ensemble --------------------------- + print("\n" + "=" * 118) + print("(A) CLOCK LENTI — TP01 daily-signal, ribilanciamento ogni N giorni " + "(tutte le fasi: min/med/max; ensemble = media dei libri sfasati)") + print("=" * 118) + clock_tbl = {} + for N in CLOCK_NS: + phase_ms, phase_nets, phase_gros = [], [], [] + for p in range(N): + pp = {a: (df, slow_clock_exec(df, tgt, N, p)) for a, (df, tgt) in pairs.items()} + m = book_eval(pp) + phase_ms.append(m) + phase_nets.append(m["net"]); phase_gros.append(m["gross"]) + all_trial_sharpes.append(m["sh_full_net"]) + ens_net = pd.concat(phase_nets, axis=1, join="inner").mean(axis=1) + ens_gro = pd.concat(phase_gros, axis=1, join="inner").mean(axis=1) + eqn = np.cumprod(1.0 + np.clip(ens_net.values, -0.99, None)) + span_y = (ens_net.index[-1] - ens_net.index[0]).total_seconds() / 86400 / 365.25 + ens = dict( + sh_full_net=al._sh(ens_net), + sh_hold_net=al._sh(ens_net[ens_net.index >= HOLDOUT]), + sh_ins_net=al._sh(ens_net[ens_net.index < HOLDOUT]), + sh_full_gro=al._sh(ens_gro), sh_hold_gro=al._sh(ens_gro[ens_gro.index >= HOLDOUT]), + maxdd=al._dd_ret(ens_net), cagr=eqn[-1] ** (1 / span_y) - 1, + turnover_y=float(np.mean([m["turnover_y"] for m in phase_ms])), + orders_y=float(np.mean([m["orders_y"] for m in phase_ms])), + net=ens_net, gross=ens_gro, + ) + all_trial_sharpes.append(ens["sh_full_net"]) + candidates[f"clock_N{N}_ensemble"] = ens + clock_tbl[N] = (phase_ms, ens) + + fn = [m["sh_full_net"] for m in phase_ms] + hn = [m["sh_hold_net"] for m in phase_ms] + fg = [m["sh_full_gro"] for m in phase_ms] + hg = [m["sh_hold_gro"] for m in phase_ms] + dd = [m["maxdd"] for m in phase_ms] + oy = [m["orders_y"] for m in phase_ms] + print(f"N={N} fasi ({N}): net FULL min/med/max {min(fn):.2f}/{np.median(fn):.2f}/{max(fn):.2f}" + f" HOLD {min(hn):.2f}/{np.median(hn):.2f}/{max(hn):.2f}" + f" | gross FULL {min(fg):.2f}/{np.median(fg):.2f}/{max(fg):.2f}" + f" HOLD {min(hg):.2f}/{np.median(hg):.2f}/{max(hg):.2f}" + f" | DD {min(dd)*100:.1f}-{max(dd)*100:.1f}% | ord/y {min(oy):.0f}-{max(oy):.0f}") + print(row(f" N={N} ENSEMBLE", ens)) + + # ---- (B) BANDE DI ISTERESI ---------------------------------------------- + print("\n" + "=" * 118) + print("(B) BANDE DI ISTERESI — decisione daily, esecuzione solo se |target−pos| > banda " + "(frazione di equity per asset); si va al target pieno") + print("=" * 118) + band_tbl = {} + for b in BANDS: + pp = {a: (df, band_exec(tgt, b)) for a, (df, tgt) in pairs.items()} + m = book_eval(pp) + band_tbl[b] = m + all_trial_sharpes.append(m["sh_full_net"]) + if b > 0: + candidates[f"band_{b:g}"] = m + saved = base["turnover_y"] - m["turnover_y"] + print(row(f"banda {b:5.3f}", m) + + f" | turn risparmiato {saved:5.1f}/y (fee ~{saved*FEE*100*2:.2f}%/y su RT)") + + # ---- selezione IN-SAMPLE (pre-2025) e hold-out riportato ----------------- + print("\n" + "=" * 118) + print("SELEZIONE CELLA — solo in-sample pre-2025 (l'hold-out si riporta, non si sceglie)") + print("=" * 118) + ranked = sorted(candidates.items(), key=lambda kv: kv[1]["sh_ins_net"], reverse=True) + for name, m in ranked: + print(f" IS {m['sh_ins_net']:5.3f} | HOLD {m['sh_hold_net']:5.3f} | FULL {m['sh_full_net']:5.3f} {name}") + chosen_name, chosen = ranked[0] + n_trials = len(all_trial_sharpes) + dsr, sr0 = al.deflated_sharpe(chosen["sh_full_net"], all_trial_sharpes, chosen["net"]) + print(f"\nCELLA SCELTA IN-SAMPLE: {chosen_name}") + print(row(" scelta", chosen)) + print(f" trials totali esplorati: {n_trials} (fasi singole incluse)") + print(f" deflated Sharpe (vs {n_trials} trial): DSR={dsr:.3f}, null-max atteso={sr0:.3f} " + f"(NB: candidato = variante di TP01, correlatissima al baseline — l'asticella " + f"giusta è lordo~uguale/netto-migliore, non earns_slot)") + dgro = chosen["sh_full_gro"] - base["sh_full_gro"] + dnet = chosen["sh_full_net"] - base["sh_full_net"] + dgro_h = chosen["sh_hold_gro"] - base["sh_hold_gro"] + dnet_h = chosen["sh_hold_net"] - base["sh_hold_net"] + print(f" Δ vs baseline — FULL: gross {dgro:+.3f} / net {dnet:+.3f} " + f"HOLD: gross {dgro_h:+.3f} / net {dnet_h:+.3f}") + print(f" fee drag baseline: turn {base['turnover_y']:.1f}/y × {2*FEE*100:.2f}%RT " + f"≈ {base['turnover_y']*FEE*100:.2f}%/y di equity — questo è il TETTO del guadagno meccanico") + + # ---- prefix-consistency (causalità dell'exec filter) --------------------- + ok = True + for a, (df, tgt) in pairs.items(): + if chosen_name.startswith("band"): + b = float(chosen_name.split("_")[1]) + full_e = band_exec(tgt, b) + cut = int(len(df) * 0.8) + sub = df.iloc[:cut].reset_index(drop=True) + sub_t = TrendPortfolio(**CANONICAL).target_series(sub) + sub_e = band_exec(sub_t, b) + elif chosen_name.startswith("clock"): + N = int(chosen_name.split("_")[1][1:]) + full_e = slow_clock_exec(df, tgt, N, 0) + cut = int(len(df) * 0.8) + sub = df.iloc[:cut].reset_index(drop=True) + sub_t = TrendPortfolio(**CANONICAL).target_series(sub) + sub_e = slow_clock_exec(sub, sub_t, N, 0) + else: + continue + d = float(np.max(np.abs(sub_e[-60:] - full_e[cut - 60:cut]))) + ok &= d < 1e-9 + print(f" prefix-consistency exec-filter (fase 0 per i clock): {'OK' if ok else 'FAIL'}") + + # ---- (6) EXECUTABILITY small-cap a 600 / 2000 / 10000 -------------------- + print("\n" + "=" * 118) + print("(6) EXECUTABILITY — min order $5, capitale per-asset = C/2 " + "(banda implicita = 5/(C/2) in peso per asset)") + print("=" * 118) + + def chosen_exec(a, df, tgt): + if chosen_name.startswith("band"): + return band_exec(tgt, float(chosen_name.split("_")[1])) + if chosen_name.startswith("clock"): + N = int(chosen_name.split("_")[1][1:]) + # deploy reale = UNA fase; usiamo fase 0 e dichiariamo la timing luck + return slow_clock_exec(df, tgt, N, 0) + return tgt.copy() + + for C in CAPITALS: + cpa = C / 2.0 + implicit = MIN_ORDER / cpa + print(f"\ncapitale ${C:.0f} (banda implicita min-order = {implicit:.4f} peso/asset)") + for label, mk in (("baseline daily", lambda a, df, t: t.copy()), + (f"variante [{chosen_name}]", chosen_exec)): + pp = {a: (df, smallcap_exec(mk(a, df, tgt), cpa)) + for a, (df, tgt) in pairs.items()} + m = book_eval(pp) + # cross-check con l'utility ufficiale (per-asset, solo full) + hc = {a: al.eval_weights_smallcap(df, mk(a, df, tgt), capital=cpa)["sharpe_haircut"] + for a, (df, tgt) in pairs.items()} + print(row(f" {label}", m) + + f" | haircut/asset vs modellato: " + + ", ".join(f"{a} {h:+.3f}" for a, h in hc.items())) + + print("\nNOTA: se la banda ottimale ≈ banda implicita a $600 (0.0167), il vincolo " + "small-cap del libro live sta GIÀ facendo il lavoro della banda.") + + +if __name__ == "__main__": + main() diff --git a/scripts/research/r0702_tp01_offset.py b/scripts/research/r0702_tp01_offset.py new file mode 100644 index 0000000..f9ce5ff --- /dev/null +++ b/scripts/research/r0702_tp01_offset.py @@ -0,0 +1,321 @@ +#!/usr/bin/env python +"""r0702_tp01_offset.py — TIMING-LUCK del ribilanciamento giornaliero di TP01. + +TP01 CANONICAL (PORT LF1d) decide sulla barra daily chiusa alle 00:00 UTC. L'ancora e' +arbitraria (Hoffstein, "rebalance timing luck"): la STESSA strategia con parametri IDENTICI +ancorata alle h:00 (h=0..23) puo' dare Sharpe diversi. Questo script: + + 1. costruisce 24 serie daily (resample 24h del 1h certificato, offset h, label/closed left, + stessa convenzione di trend_portfolio.resample_tf) — SANITY OBBLIGATORIO: h=0 riproduce + ESATTAMENTE al.tp01_baseline_daily() (stesso Sharpe FULL/HOLD); + 2. misura Sharpe/CAGR/maxDD FULL, IS (pre-2025) e HOLD-OUT per offset -> percentile di h=0; + 3. ENSEMBLE (tranching 1/K del capitale per ancora): 24-way + K=2 (0,12), K=3 (0,8,16), + K=4 (0,6,12,18) — scelte A PRIORI simmetriche, zero tuning per-offset, zero selezione + sull'hold-out. L'ensemble e' valutato sul BOOK AGGREGATO su griglia 1h (posizione = + media delle tranche, fee sul turnover netto reale) — non media di equity separate; + 4. dispersione: std/range dello Sharpe fra le 24 ancore vs fra TUTTE le rotazioni possibili + di K=2 (12), K=3 (8), K=4 (6) — la riduzione di varianza e' il criterio strutturale; + 5. small-cap: haircut min-order $5 a capitale 600/2k/10k per K=1 vs K=2/4/24 + (il tranching divide gli ordini per K -> piu' rebalance sotto min-order). + +Causalita': targets TP01 causali per costruzione; guardia ricalcolo-su-prefisso sia sul +daily resampled sia sul troncamento del 1h; mappatura daily->1h via merge_asof backward su +EPOCA MS ESPLICITA (mai .view su tz-aware); eval_weights fa lo shift (held durante t+1). + +Vincoli: nessun file toccato fuori da questo script. Runtime ~1-2 min. +""" +from __future__ import annotations + +import sys +from functools import lru_cache +from pathlib import Path + +import numpy as np +import pandas as pd + +ROOT = Path("/opt/docker/PythagorasGoal") +sys.path.insert(0, str(ROOT / "scripts" / "research" / "alt")) +sys.path.insert(0, str(ROOT)) + +import altlib as al # noqa: E402 +from src.strategies.trend_portfolio import CANONICAL, TrendPortfolio # noqa: E402 + +TP = TrendPortfolio(**CANONICAL) +HOLDOUT = al.HOLDOUT +ASSETS = ("BTC", "ETH") +MS_H = 3_600_000 +MS_D = 86_400_000 + +# ancore a priori, simmetriche, NON ottimizzate +HEADLINE = { + "K=1 (canonico h=0)": (0,), + "K=2 (0,12)": (0, 12), + "K=3 (0,8,16)": (0, 8, 16), + "K=4 (0,6,12,18)": (0, 6, 12, 18), + "K=24 (tutte)": tuple(range(24)), +} + + +# --------------------------------------------------------------------------- +# Resample con ancora h — stessa convenzione di trend_portfolio.resample_tf +# --------------------------------------------------------------------------- +def resample_offset(df_1h: pd.DataFrame, h: int) -> pd.DataFrame: + g = df_1h.copy() + idx = pd.to_datetime(g["timestamp"], unit="ms", utc=True) + idx.name = "dt" + g.index = idx + out = g.resample("24h", offset=pd.Timedelta(hours=h), label="left", closed="left").agg( + {"open": "first", "high": "max", "low": "min", "close": "last", "volume": "sum"}) + out = out.dropna(subset=["open"]) + out["datetime"] = out.index + epoch = pd.Timestamp("1970-01-01", tz="UTC") + out["timestamp"] = ((out.index - epoch) // pd.Timedelta(milliseconds=1)).astype("int64") + return out.reset_index(drop=True)[ + ["timestamp", "open", "high", "low", "close", "volume", "datetime"]] + + +@lru_cache(maxsize=8) +def get1h(asset: str) -> pd.DataFrame: + return al.get(asset, "1h") + + +@lru_cache(maxsize=64) +def daily_off(asset: str, h: int) -> pd.DataFrame: + return resample_offset(get1h(asset), h) + + +# --------------------------------------------------------------------------- +# Metriche su serie daily (convenzione identica al baseline: _to_daily + _sh) +# --------------------------------------------------------------------------- +def dmetrics(s: pd.Series) -> dict: + s = s.dropna() + is_ = s[s.index < HOLDOUT] + ho = s[s.index >= HOLDOUT] + eq = float(np.prod(1.0 + s.values)) + yrs = len(s) / 365.25 + cagr = eq ** (1 / yrs) - 1 if eq > 0 and yrs > 0 else -1.0 + return dict(sh_full=al._sh(s), sh_is=al._sh(is_), sh_hold=al._sh(ho), + cagr=cagr, dd=al._dd_ret(s), dd_hold=al._dd_ret(ho), n=len(s)) + + +# --------------------------------------------------------------------------- +# Path DAILY-NATIVO per offset (counterfactual "e se l'ancora fosse h") +# --------------------------------------------------------------------------- +def port_daily_native(h: int) -> pd.Series: + series = {} + for a in ASSETS: + df = daily_off(a, h) + net, ts = TP.net_returns(df) + series[a] = pd.Series(net, index=pd.DatetimeIndex(pd.to_datetime(ts.values, utc=True))) + J = pd.concat(series, axis=1, join="inner").fillna(0.0) + return al._to_daily(0.5 * J[ASSETS[0]] + 0.5 * J[ASSETS[1]]) + + +# --------------------------------------------------------------------------- +# Path 1h AGGREGATO (book unico): target daily-offset mappato causale sul 1h +# --------------------------------------------------------------------------- +@lru_cache(maxsize=64) +def hourly_target(asset: str, h: int) -> tuple: + """Target TP01 (ancora h) sul grid 1h: per ogni barra 1h prendi il target dell'ultima + barra daily-offset il cui CLOSE nominale (label+24h, epoca ms) e' <= close della barra 1h + (ts+1h). merge_asof backward su int ms espliciti. eval_weights poi SHIFTA (held t+1).""" + d = daily_off(asset, h) + tgt = TP.target_series(d) + right = pd.DataFrame({"cms": d["timestamp"].values.astype("int64") + MS_D, + "tgt": tgt}) + left = pd.DataFrame({"cms": get1h(asset)["timestamp"].values.astype("int64") + MS_H}) + m = pd.merge_asof(left, right, on="cms", direction="backward") + return tuple(np.nan_to_num(m["tgt"].values, nan=0.0)) + + +def ens_target(asset: str, hs: tuple) -> np.ndarray: + return np.mean([np.asarray(hourly_target(asset, h)) for h in hs], axis=0) + + +def port_hourly(hs: tuple) -> tuple[pd.Series, float]: + """Serie daily del book aggregato (0.5 BTC + 0.5 ETH su grid 1h) + turnover/anno.""" + nets, turns = {}, 0.0 + for a in ASSETS: + df = get1h(a) + ev = al.eval_weights(df, ens_target(a, hs)) + nets[a] = pd.Series(ev["net"], index=ev["idx"]) + turns += 0.5 * ev["turnover_per_year"] + J = pd.concat(nets, axis=1, join="inner").fillna(0.0) + return al._to_daily(0.5 * J[ASSETS[0]] + 0.5 * J[ASSETS[1]]), turns + + +# --------------------------------------------------------------------------- +# Small-cap: min-order $5, capitale condiviso 50/50 fra i 2 asset +# --------------------------------------------------------------------------- +def smallcap_net(df: pd.DataFrame, tgt: np.ndarray, capital: float, + min_order: float = 5.0) -> tuple[pd.Series, int]: + """Copia locale della logica di al.eval_weights_smallcap che restituisce la serie net + (serve per combinare il book 2-asset). Cambi di nozionale < min_order NON eseguiti.""" + c = df["close"].values.astype(float) + tgt = np.clip(np.nan_to_num(np.asarray(tgt, float)), -10, 10) + held = np.empty(len(tgt)) + cur, n_tr = 0.0, 0 + for i in range(len(tgt)): + if abs(tgt[i] - cur) * capital >= min_order: + cur = tgt[i] + n_tr += 1 + held[i] = cur + r = al.simple_returns(c) + pos = np.zeros(len(held)) + pos[1:] = held[:-1] + turn = np.abs(np.diff(pos, prepend=0.0)) + net = pos * r - al.FEE_SIDE * turn + net[0] = 0.0 + idx = pd.DatetimeIndex(pd.to_datetime(df["datetime"], utc=True)) + return pd.Series(net, index=idx), n_tr + + +def smallcap_port(hs: tuple, capital: float) -> dict: + """Book realistico a `capital`: target per-asset = 0.5*ensemble (quota 50/50). + modeled = stesso book senza vincolo min-order (fee identiche proporzionali).""" + nets_r, nets_m, ntr = {}, {}, 0 + for a in ASSETS: + df = get1h(a) + t = 0.5 * ens_target(a, hs) + nr, n = smallcap_net(df, t, capital) + nets_r[a] = nr + ntr += n + ev = al.eval_weights(df, t) + nets_m[a] = pd.Series(ev["net"], index=ev["idx"]) + Jr = pd.concat(nets_r, axis=1, join="inner").fillna(0.0) + Jm = pd.concat(nets_m, axis=1, join="inner").fillna(0.0) + dr = al._to_daily(Jr[ASSETS[0]] + Jr[ASSETS[1]]) + dm = al._to_daily(Jm[ASSETS[0]] + Jm[ASSETS[1]]) + yrs = len(dr) / 365.25 + return dict(sh_real=al._sh(dr), sh_model=al._sh(dm), + haircut=al._sh(dm) - al._sh(dr), + dd_real=al._dd_ret(dr), trades_per_year=ntr / yrs) + + +# --------------------------------------------------------------------------- +# Guardie +# --------------------------------------------------------------------------- +def sanity_h0() -> None: + """h=0 deve riprodurre ESATTAMENTE il baseline (dati + strategia + metriche).""" + for a in ASSETS: + d0 = daily_off(a, 0) + ref = al.get(a, "1d") + for col in ("timestamp", "open", "high", "low", "close", "volume"): + assert np.allclose(d0[col].values.astype(float), + ref[col].values.astype(float), atol=0, rtol=0), \ + f"resample_offset(h=0) != al.get('{a}','1d') su {col}" + mine = port_daily_native(0) + base = al.tp01_baseline_daily() + assert len(mine) == len(base) and np.allclose(mine.values, base.values, atol=1e-12), \ + "portafoglio h=0 != tp01_baseline_daily" + mm, bb = dmetrics(mine), dmetrics(base) + print(f"[SANITY] h=0 == baseline: OK (Sharpe FULL {mm['sh_full']:.4f} == " + f"{bb['sh_full']:.4f}, HOLD {mm['sh_hold']:.4f} == {bb['sh_hold']:.4f})") + + +def causality_guards() -> None: + """(a) prefix-recompute sul daily resampled: target[i] non cambia aggiungendo futuro. + (b) troncando il 1h, le barre daily complete restano identiche (solo l'ultima e' parziale).""" + for a in ASSETS: + for h in (0, 7, 13, 21): + d = daily_off(a, h) + t_full = TP.target_series(d) + cut = len(d) - 250 + t_pref = TP.target_series(d.iloc[:cut].reset_index(drop=True)) + assert np.allclose(t_full[:cut], t_pref, atol=1e-12), \ + f"prefix-recompute FAIL {a} h={h}" + df1h = get1h(a) + d_tr = resample_offset(df1h.iloc[:-500].reset_index(drop=True), h) + k = len(d_tr) - 1 # l'ultima barra del troncato e' parziale per costruzione + for col in ("timestamp", "close"): + assert np.allclose(d_tr[col].values[:k].astype(float), + d[col].values[:k].astype(float)), \ + f"1h-truncation FAIL {a} h={h} {col}" + print("[SANITY] guardie causalita' (prefix-recompute daily + troncamento 1h): OK") + + +# --------------------------------------------------------------------------- +# Main +# --------------------------------------------------------------------------- +def main() -> None: + print("=" * 96) + print("r0702 — TP01 rebalance timing-luck: 24 ancore daily + ensemble tranching") + print(f"CANONICAL={CANONICAL} fee 0.10% RT HOLD-OUT >= {HOLDOUT.date()}") + print("=" * 96) + + sanity_h0() + causality_guards() + + # ---- (1) per-offset, path daily-nativo -------------------------------- + rows = [] + for h in range(24): + m = dmetrics(port_daily_native(h)) + rows.append(dict(h=h, **m)) + T = pd.DataFrame(rows).set_index("h") + + print("\n--- (1) PER-OFFSET (daily nativo, ancora h:00 UTC) ---") + print(f"{'h':>3} {'ShFULL':>7} {'ShIS':>7} {'ShHOLD':>7} {'CAGR':>7} {'maxDD':>7}") + for h, r in T.iterrows(): + tag = " <- canonico" if h == 0 else "" + print(f"{h:>3} {r.sh_full:>7.3f} {r.sh_is:>7.3f} {r.sh_hold:>7.3f} " + f"{r.cagr:>6.1%} {r.dd:>6.1%}{tag}") + + def pctl(col: str) -> float: + v = T[col].values + return float((v < v[0]).mean() + 0.5 * (v == v[0]).mean()) * 100 + + print("\nDistribuzione fra le 24 ancore (min / mediana / max / std) [percentile di h=0]:") + for col, lbl in (("sh_full", "Sharpe FULL"), ("sh_is", "Sharpe IS(pre-2025)"), + ("sh_hold", "Sharpe HOLD"), ("dd", "maxDD"), ("cagr", "CAGR")): + v = T[col] + print(f" {lbl:<20} {v.min():>7.3f} / {v.median():>7.3f} / {v.max():>7.3f} " + f"/ std {v.std():.3f} h=0 al {pctl(col):.0f}° pctl (val {v.iloc[0]:.3f})") + + # ---- (2) ensemble headline, book aggregato su 1h ---------------------- + print("\n--- (2) ENSEMBLE (book aggregato su grid 1h, fee su turnover netto) ---") + print(f"{'config':<22} {'ShFULL':>7} {'ShIS':>7} {'ShHOLD':>7} {'CAGR':>7} " + f"{'maxDD':>7} {'DDhold':>7} {'turn/y':>7}") + head = {} + for name, hs in HEADLINE.items(): + s, tpy = port_hourly(hs) + m = dmetrics(s) + head[name] = m + print(f"{name:<22} {m['sh_full']:>7.3f} {m['sh_is']:>7.3f} {m['sh_hold']:>7.3f} " + f"{m['cagr']:>6.1%} {m['dd']:>6.1%} {m['dd_hold']:>6.1%} {tpy:>7.1f}") + print("(nota: 'K=1 (canonico h=0)' qui e' lo stesso book valutato sul grid 1h — " + "differenze vs riga h=0 sopra = sola granularita' di compounding, non strategia)") + + # ---- (3) varianza della stima: rotazioni complete per famiglia -------- + print("\n--- (3) DISPERSIONE fra rotazioni (nessuna selezione: TUTTE le rotazioni) ---") + fams = { + "singole (24)": [(h,) for h in range(24)], + "K=2 h,h+12 (12)": [(h, h + 12) for h in range(12)], + "K=3 h,h+8,h+16 (8)": [(h, h + 8, h + 16) for h in range(8)], + "K=4 h,h+6,.. (6)": [tuple(h + 6 * j for j in range(4)) for h in range(6)], + } + print(f"{'famiglia':<20} {'ShFULL μ':>9} {'σ':>6} {'range':>13} " + f"{'ShHOLD μ':>9} {'σ':>6} {'range':>13}") + for name, rot in fams.items(): + mf = [dmetrics(port_hourly(hs)[0]) for hs in rot] + f = np.array([m["sh_full"] for m in mf]) + ho = np.array([m["sh_hold"] for m in mf]) + print(f"{name:<20} {f.mean():>9.3f} {f.std():>6.3f} " + f"[{f.min():>5.3f},{f.max():>5.3f}] " + f"{ho.mean():>9.3f} {ho.std():>6.3f} [{ho.min():>5.3f},{ho.max():>5.3f}]") + + # ---- (4) small-cap: haircut min-order per capitale -------------------- + print("\n--- (4) SMALL-CAP (min order $5, capitale 50/50 sui 2 asset) ---") + print(f"{'config':<22} {'cap':>7} {'Sh model':>9} {'Sh real':>8} {'haircut':>8} " + f"{'DD real':>8} {'trade/y':>8}") + for name, hs in HEADLINE.items(): + for cap in (600, 2000, 10000): + r = smallcap_port(hs, cap) + print(f"{name:<22} {cap:>7} {r['sh_model']:>9.3f} {r['sh_real']:>8.3f} " + f"{r['haircut']:>8.3f} {r['dd_real']:>7.1%} {r['trades_per_year']:>8.1f}") + + print("\nFatto. Nessuna selezione sull'hold-out: ensemble a priori, giudizio su " + "struttura (varianza) + IS pre-2025; l'hold-out e' solo riportato.") + + +if __name__ == "__main__": + main() diff --git a/src/strategies/trend_portfolio.py b/src/strategies/trend_portfolio.py index 5914cbe..a5c9515 100644 --- a/src/strategies/trend_portfolio.py +++ b/src/strategies/trend_portfolio.py @@ -7,6 +7,11 @@ fee 0.10% RT, positiva ogni anno 2019-2026, robusta su griglia e su tutti i time Config canonica deployabile (PORT LF1d): timeframe >=12h (1d RACCOMANDATO), LONG-FLAT (niente short), vol-target 20%, leverage cap 2x. -> FULL Sharpe ~1.30, maxDD ~14%, HOLD-OUT 2025-26 Sharpe ~0.31 (calcolo per-TF leak-free). + NB ANCHOR TIMING-LUCK (2026-07-02): lo 0.31 e' sull'ancora daily 00:00 UTC, la MIGLIORE + delle 24 possibili (mediana ancore 0.04, banda [-0.13,+0.30], P~0.86 di uno spike cosi' + per caso) -> l'hold-out non risolve l'edge di RITORNO; cio' che regge a OGNI ancora e' il + taglio del DD (7-10% vs ~60% B&H). FULL/plateau/gate invariati (h=0 al 31mo pctl su FULL). + Vedi docs/diary/2026-07-02-timing-crt-wave.md e scripts/research/r0702_tp01_offset.py. NB LOOK-AHEAD (2026-06-19): un ffill MIXED-TIMEFRAME su barre open-labeled (label="left") gonfiava il 4h (~1.60 -> reale ~1.1). Il calcolo per-SINGOLO-TF e' leak-free (guard