diff --git a/docs/diary/2026-06-19-cerbero-bite-mainnet-verified.md b/docs/diary/2026-06-19-cerbero-bite-mainnet-verified.md new file mode 100644 index 0000000..5758133 --- /dev/null +++ b/docs/diary/2026-06-19-cerbero-bite-mainnet-verified.md @@ -0,0 +1,65 @@ +# 2026-06-19 — Cerbero-bite = MAINNET reale: fonte VRP sbloccata + +Indagine "cerca dati di cerbero-bite" + verifica mainnet/testnet a tre livelli. Esito: la +contaminazione storica NON era una proprieta' di Cerbero MCP, ma del vecchio token testnet sul +solo endpoint `get_historical`. Il token di cerbero-bite e' mainnet e serve catene opzioni reali. + +## Dove sono i dati di cerbero-bite + +`/home/adriano/Documenti/Git_XYZ/CerberoSuite/Cerbero_Bite` — bot live (testnet exec, propose-only) +che vende **credit-spread bull-put su ETH**. Dati: +- `data/state.sqlite`: `market_snapshots` (**52 righe, solo 30 apr–1 mag 2026**, BTC+ETH) con + `spot, dvol, realized_vol_30d, iv_minus_rv, funding_perp/cross, dealer_net_gamma, + gamma_flip_level, oi_delta_pct_4h, liquidation_long/short_risk, macro_days_to_event`; + `dvol_history` (1 riga); `positions/instructions/decisions` (0 righe, niente trade persistiti). +- `data/log/*.jsonl` (26 apr–1 mag 2026): log HTTP, non dump di catena. `strategy.yaml`: golden config. +- **Fonte dati**: Cerbero MCP (`get_instruments` + `get_ticker_batch`) dal gateway + `cerbero-mcp.tielogic.xyz`. NON c'e' storico profondo della catena (solo fetch live/on-demand). + +## Verifica mainnet vs testnet (3 livelli) + +1. **Spot vs nostra serie certificata** (Deribit mainnet), 2026-04-30 13–16h UTC: + BTC cerbero 76.287–76.446 vs certificato 76.237–76.443 (Δ 0.13–0.27%); ETH 2.261–2.264 vs + 2.256–2.265 (Δ 0.04–0.29%). Scarti = rumore intra-barra (snapshot 15-min vs close orario). + NON e' il feed fantasma testnet (che divergeva >3%). +2. **`environment_info`** (token cerbero-bite): `environment=mainnet`, `base_url=www.deribit.com`, + `source=credentials`. **`get_ticker ETH-PERPETUAL`**: `testnet=false`, mark 1703.11. +3. **Catena, decisivo** — stessa opzione su ccxt.deribit mainnet vs Cerbero MCP: + `ETH_USDC-26JUN26-1650-P` (put settimanale, delta ~-0.28): + + | fonte | bid | ask | mark_iv | delta | testnet | + |---|---|---|---|---|---| + | ccxt mainnet | 25.6 | 26.6 | 54.54% | -0.3150 | — | + | Cerbero MCP | 25.6 | 26.6 | 54.54% | -0.31513 | False | + + **Identici bit-per-bit.** + +## Verdetto + +- **Il token MCP di cerbero-bite e' MAINNET; la sua catena opzioni e' reale** (= ccxt.deribit + mainnet). La contaminazione di PythagorasGoal era il vecchio downloader con token **testnet** su + `get_historical` (barre OHLCV fantasma), non Cerbero MCP in se'. +- **Fonte VRP sbloccata**: Cerbero MCP da' bid/ask/IV/greche/OI per-strike (come ccxt) **+** feature + di regime che ccxt non ha (`dealer_net_gamma`, `gamma_flip_level`, `oi_delta_pct_4h`, + `liquidation_*`, `funding`, `iv_minus_rv`, `macro`). Utile per validare lo sleeve VRP su piu' + regimi (raccolta snapshot live + accumulo nel tempo). +- **Limite residuo**: niente storico profondo della catena -> il backtest pluriennale del VRP resta + prezzato da modello (DVOL+BS); ma la calibrazione model-vs-reale e' ora robusta e ripetibile + (snapshot reali su piu' date/regimi). + +## Collegamento col lavoro VRP (sleeve opzioni) + +Conferma e rafforza `2026-06-19-eval-crypto-backtest-options.md`: lo snapshot ccxt aveva gia' +mostrato che il backtest SOTTOSTIMA il premio (skew +28% > spread 4% -> bid reale = 1.29x modello). +Ora abbiamo due fonti mainnet concordi (ccxt + Cerbero MCP) per misurare premio/skew/spread su piu' +regimi. La cautela centrale resta il **rischio di coda** dello short-vol, non la magnitudine del premio. + +## Stato cerbero-bite (gia' concluso, contesto) + +Il credit-spread bull-put ETH e' gia' stato giudicato NON robusto su ciclo completo (diario +`Old/docs/diary/2026-06-09-cerbero-bite-credit-spread.md`: EV breakeven-negativo; "+0.48%/mese" = +artefatto di finestra calma; coda concentrata col fade ETH). E' una struttura diversa dalla +put-selling/wheel del progetto `crypto_backtest`. + +> Sicurezza: il token di cerbero-bite e' stato usato solo per la verifica; mai stampato ne' committato +> (resta in `.env`, gitignored). diff --git a/docs/diary/2026-06-19-eval-crypto-backtest-options.md b/docs/diary/2026-06-19-eval-crypto-backtest-options.md new file mode 100644 index 0000000..567989a --- /dev/null +++ b/docs/diary/2026-06-19-eval-crypto-backtest-options.md @@ -0,0 +1,125 @@ +# 2026-06-19 — Valutazione strategia esterna `crypto_backtest` (trend + opzioni VRP) + +Valutazione critica di un progetto esterno (`/home/adriano/crypto_backtest/`, file chiave +`STRATEGIA.md`, `production.py`, `options_deribit.py`, `production_equity.csv`) che propone un +book a 2 motori quasi scorrelati. Rilevante perché tocca proprio la frontiera che la nostra +ricerca post-reset ha lasciato aperta (le opzioni / volatility risk premium). + +## Cosa propone + +Portafoglio a due gambe (ρ=0.22 verificato dal CSV): +- **Sleeve 1 (25%)** — trend spot BTC+ETH a **12h**, long-only se `trend(30g)>0`, vol-target 20%, + cap 3×, leva globale ~1.07 calibrata a maxDD in-sample −20%. +- **Sleeve 2 (75%)** — vendita di **put settimanali (CSP/wheel) su BTC** su Deribit, strike a + **delta 0.28**, hold-to-expiry, IV da DVOL reale, prezzo Black-Scholes. + +Numeri riprodotti dal CSV (finestra 2021-04→2026-06, 272 settimane): + +| Serie | CAGR | Sharpe | maxDD | final | +|---|---|---|---|---| +| spot | +12.0% | 0.77 | −18.1% | 1.80x | +| opt | +15.9% | 1.09 | −20.0% | 2.16x | +| **blend 25/75** | +15.4% | **1.21** | **−15.2%** | 2.10x | +| blend ri-levato | +20.5% | 1.21 | −20.0% | 2.63x | +| B&H BTC | +1.3% | 0.30 | −74.2% | 1.07x | + +corr(spot, opt) = **0.217** confermata. Settimane peggiori opt: 2022-05 (LUNA) −13%, +2022-06 −11%, 2021-05 −11%, 2022-11 (FTX) −9.7%. + +## Punto forte — corroborazione indipendente del nostro TP01 + +Lo **sleeve spot è quasi identico al nostro TP01** (`src/strategies/trend_portfolio.py`): +12h, long-only, trend(30g), vol-target 20%, cap 3×. Due ricerche separate, due dataset diversi +(loro Binance, noi Deribit certificato), **stessa conclusione**: il trend vol-targeted a 12h è +l'edge reale e robusto. Il nostro Sharpe è più alto (1.32 vs 0.77 su questa finestra / 1.07 +full-history) perché usiamo un **blend multi-orizzonte 1-3-6m** invece del singolo trend a 30g → +il blend diversifica gli orizzonti e alza lo Sharpe. Conferma forte per entrambi. + +NB: loro confermano anche le NOSTRE lezioni — intraday ≤1h scartato (costi/rumore), un **bug di +look-ahead sul 4h trovato e corretto** (identico al nostro audit), MR/condor/strangle nudi e +collar stretti scartati per overfit/tail. + +## Punto critico — lo sleeve opzioni guida il 75% ma è prezzato dal proprio modello + +È esattamente il muro che avevamo dichiarato non-backtestabile (W18/19/21, ARGO: niente storico +chain per-strike gratis). Il loro workaround (BS su **DVOL reale** + payoff sul path realizzato) +fa emergere il VRP perché IV>RV (misurato BTC IV/RV~1.24). Concettualmente sano, ma la +**magnitudine è ottimistica** — limiti (in parte ammessi dagli autori): + +1. **Nessun bid/ask**: vendono al mid (BS fair), non al bid. Sulle put OTM settimanali lo spread + è grosso → premio reale nettamente inferiore. +2. **Skew ignorato**: prezzano put a delta-0.28 (OTM) con DVOL = **IV ATM**. Il mercato carica le + put molto di più (skew di crash) → modellano la vol sbagliata proprio sull'opzione venduta. +3. **Coda sotto-modellata**: settimana peggiore solo −13% attraverso LUNA/FTX → sospettosamente + benigno per un venditore di put nudo. Gap, illiquidità di roll e settlement inverso (coin-settled) + sono approssimati. +4. **Leva senza funding** (ottimistico) + **bias di finestra** (parte vicino al top 2021, + favorevole a un book short-vol DD-capped). + +Il blend Sharpe 1.21 è dominato dallo sleeve income (Sharpe 1.09, peso 75%). Con bid/ask + skew + +coda realistica lo sleeve income vale plausibilmente molto meno (Sharpe reale stimato ~0.7-0.9), +e il blend scende di conseguenza. + +## Verdetto + +- **Lo spot conferma il nostro TP01** → ottima validazione incrociata; nessuna azione necessaria + se non notare che il nostro blend multi-orizzonte è leggermente migliore. +- **Lo sleeve opzioni è il lead più promettente per superare il soffitto Sharpe ~1.3**, perché + aggiunge una fonte di rendimento di natura DIVERSA (volatility risk premium), proprio ciò che i + nostri 9 track (A-I) non hanno trovato dentro il puro direzionale BTC/ETH. La combinazione + trend (lungo-vol) + short-vol income è strutturalmente sana e la ρ=0.22 è reale. +- **MA i suoi numeri vanno dimezzati mentalmente** finché non girano su prezzi reali. Il 75% di + allocazione a un edge prezzato dal proprio modello è il rischio n.1. + +## Prossimi passi onesti se si vuole inseguire questo lead + +1. **Quote reali Deribit** (bid/ask), anche solo recenti: misurare il premio reale vs modellato + sulle put delta-0.28 settimanali, e quanto Sharpe sopravvive allo spread. +2. **Prezzare allo skew vero** (IV della put OTM, non DVOL ATM). +3. **Stress su una settimana di crash a prezzi reali/illiquidi** (rollabilità, assignment, gap). +4. **Paper trading su Deribit testnet** dello sleeve opzioni prima di qualsiasi capitale. + +Coerente con la regola del progetto (lezione v2.0.0): un edge full+OOS robusto su prezzi MODELLATI +non è un edge finché non è verificato su prezzi reali ed eseguibili. + +--- + +## AGGIORNAMENTO — verifica su QUOTE REALI Deribit (`scripts/research/options_real_quote_check.py`) + +Fatta la verifica concreta (PARTE 1: catena reale Deribit mainnet pubblico; PARTE 2: ri-esecuzione +dello sleeve CSP con haircut reale sul premio). **Risultato che RIBALTA una mia critica.** + +Snapshot del 2026-06-19, scadenza settimanale 2026-06-26 (~6.2 DTE), put delta −0.277 (strike 61k, +3.1% OTM), underlying 62.965: + +| Grandezza | Valore | +|---|---| +| IV ATM (≈ DVOL) | 37.2% | +| IV put OTM (mark) | 42.1% (**skew +4.8 pt**) | +| premio put: BID / mark / ask | 598 / 623 / 630 USD | +| spread bid/mark | 0.96 (spread ~4%) | +| premio MODELLATO dal backtest (BS @ IV-ATM) | **463 USD** | +| **HAIRCUT premio reale(BID)/modello** | **1.29** | + +**Il backtest SOTTOSTIMA il premio, non lo sovrastima.** Prezzando la put OTM con la DVOL (IV ATM) +ignora lo skew (+28% sul premio lordo); il bid/ask la riporta giu' solo del 4% → vendendo al BID +reale incassi **1.29×** il premio modellato. Lo sleeve modellato (Sharpe 1.13) e' quindi +**conservativo sul premio** alle quote attuali; col premio reale salirebbe (Sharpe → 1.83 a f=1.29). + +**Ma la critica vera si SPOSTA, non sparisce:** lo skew esiste perche' il mercato prezza la coda +grassa: piu' premio = esattamente perche' i crash fanno male. La sensitivity mostra il punto di +rottura — lo sleeve regge finche' incassi >~85% del premio modellato (Sharpe 0.59 a f=0.85), va a +zero a f=0.70, negativo a f=0.55. Lo snapshot e' in **regime calmo** (IV ATM 37%, bassa per crypto); +in un crash lo spread si allarga molto e potresti non riuscire a rollare. Quindi: + +- ✅ **Concern "premio sovrastimato" = SMENTITO** (alle quote attuali e' anzi sottostimato). +- ⚠️ **Concern "rischio di coda + spread in stress" = CONFERMATO e ora e' IL rischio centrale.** + Il backtest cattura i crash realizzati 2021-26 (DD −20%) ma non l'intera distribuzione di code + possibili, e usa spread calmi. La f reale in settimana di crash e' < 1 e lo spread esplode. + +**Verdetto aggiornato:** lo sleeve income e' piu' solido di quanto temessi sul *premio* (il VRP + +skew e' reale e generoso), ma resta una strategia short-vol il cui rischio vero e' la **coda** e la +**liquidita' di roll nello stress**, non la magnitudine del premio. Prima del capitale: ripetere lo +snapshot nel tempo (specie in regimi di IV alta), misurare lo spread in giornate di stress, e +paper-trade su testnet. Il lead per superare il soffitto Sharpe ~1.3 (aggiungere il VRP a TP01) +resta valido e ora meglio quantificato. diff --git a/docs/diary/2026-06-19-trackF-seasonality.md b/docs/diary/2026-06-19-trackF-seasonality.md new file mode 100644 index 0000000..7415628 --- /dev/null +++ b/docs/diary/2026-06-19-trackF-seasonality.md @@ -0,0 +1,77 @@ +# Track F — Calendar seasonality (hour-of-day / day-of-week) on BTC & ETH + +**Data:** 2026-06-19 · **Script:** `scripts/research/trackF_seasonality.py` +**Dati:** Deribit mainnet certificati, BTC/ETH 1h UTC. Fee baseline 0.10% RT (`fee_side=0.0005`). + +## Domanda +Esiste un edge di calendario *sistematico e tradeable* (ora del giorno, giorno della +settimana, interazione ora×giorno) su BTC ed ETH, netto fee, OOS, per-anno, su entrambi gli asset? + +## Metodologia (anti-overfit, anti-leakage) +- `ret[i]=close[i]/close[i-1]-1` è noto a `close[i]`; una posizione decisa a `close[i]` guadagna + `ret[i+1]`. La statistica che decide il trade usa **solo barre ≤ i** (mai la barra tradata né futuro). +- **Tradeable test onesto = ADAPTIVE EXPANDING sign**: a `close[i]` guardo il bucket di calendario + della barra `i+1` (il clock è noto, zero look-ahead) e prendo il **segno della media passata** di + quel bucket (espandente, warmup-gated). Long-flat o long-short. Fee solo su `|Δposizione|`. + È l'analogo onesto di "tradare il seasonal": i dati scelgono il segno di ogni bucket **dal vivo**. +- Tabelle descrittive per-ora/per-giorno split IS(65%)/OOS(35%) come diagnostica. +- Regola discreta ottimizzata in-sample (entra a ora H, tieni W barre, dir migliore) mostrata solo + per **esporre il gap IS→OOS** (384 celle testate/asset). +- Benchmark **buy-and-hold** come controllo del long-bias. + +## Risultati + +### 1. Descrittive (bp/barra, IS vs OOS) +- **Hour-of-day:** sign-agreement IS/OOS solo **12/24 (BTC)** e **8/24 (ETH)** → caso. Le ore "US + close" 21:00–22:00 UTC sono positive in entrambi gli split su entrambi gli asset (l'unico pattern + con un minimo di coerenza), ma il resto è rumore che cambia segno tra IS e OOS. +- **Day-of-week:** più stabile. **Giovedì negativo** su BTC ed ETH in IS *e* OOS; Lun/Mer positivi. + Sign-agreement 6/7 (BTC), 5/7 (ETH). + +### 2. Adaptive expanding-sign (il test tradeable) +| Strategia | BTC Sharpe | ETH Sharpe | Note | +|---|---|---|---| +| HOUR long-short | **−5.39** | **−4.04** | DD 100%. Annientata dalle fee. | +| HOUR long-flat | −2.92 | −2.09 | DD 100%. Idem. | +| DOW long-short | +0.64 | +0.83 | DD 82–84%, −66% nel 2022 | +| DOW long-flat | +0.81 | +0.96 | DD 75–78%, −64/−66% nel 2022 | +| HOUR×WEEKDAY (168 buckets) | −5.05 | −3.96 | DD 100%. Overfit puro + fee. | + +### 3. Il controllo che smonta il DOW — **buy-and-hold** +- BTC buy-hold: **Sharpe 0.79, CAGR 34.9%, DD 77%** → DOW long-flat: Sh 0.81, CAGR 34.2%, DD 77.5%. +- ETH buy-hold: **Sharpe 0.84, CAGR 42.4%, DD 81%** → DOW long-flat: Sh 0.96, CAGR 52.7%, DD 74%. +- Il DOW long-flat è **long il 78% del tempo** (`mean_pos≈+0.78`). È **buy-and-hold travestito**: + guadagna perché crypto sale, non perché esiste un edge di giorno. Lo "skip del giovedì" aggiunge + pochissimo e non giustifica un deploy. + +### 4. Fee sweep (HOUR long-short adaptive) +A fee **0%**: Sh +0.61 (BTC) / +0.80 (ETH) — solo long-drift. A 0.10% RT: **−5.4 / −4.0**. Turnover +**~8.000 flip/anno** (segno orario instabile, cambia quasi ogni barra) → morte istantanea per fee. +Le strategie hour-of-day sono ad alta frequenza per costruzione: le fee sono di prim'ordine e le +uccidono. + +### 5. Regola discreta ottimizzata in-sample (trappola multiple-testing) +- BTC: best IS H=05 hold=24h dir=+1 → **IS Sh +4.25 → OOS Sh +1.47** (+3.7 bp/trade). +- ETH: best IS H=13 hold=24h dir=+1 → **IS Sh +7.35 → OOS Sh +0.90** (+3.2 bp/trade). +- Collasso IS→OOS classico. Inoltre "hold 24h dir+1" = ancora **long-bias** (entra una volta/giorno + e tiene 24h ≈ sempre long). Il margine OOS (~3 bp/trade su 10 bp RT) è marginale e fragile. + +## Multiple-testing +199 celle di calendario/asset (24 ore + 7 giorni + 168 ora×giorno) + 384 (H,W,dir)/asset. Con così +tante celle, bucket "significativi" spuri sono **garantiti**. Filtri applicati: segno scelto dal vivo +su soli dati passati, deve reggere OOS, per-anno, e su **entrambi** BTC ed ETH. + +## Verdetto — **SPURIO / NON deployable** +- **Nessun edge di calendario netto-fee robusto** su BTC ed ETH. +- **Hour-of-day:** morto (fee + segno instabile). L'unica regolarità (US-close 21–22 UTC positiva) è + troppo debole e non sopravvive al turnover. +- **Day-of-week:** l'unico risultato "positivo" è **long-bias mascherato** (≈ buy-and-hold, + Sharpe ~0.8–0.96 < trend portfolio 1.32, DD 75–84% rovinoso, −65% nel 2022). Non è un edge + seasonal sfruttabile; è esposizione direzionale al drift di crypto. +- **Hour×weekday:** overfit puro (IS −3.6 → OOS −8.0). +- Coerente con la lezione del progetto: dove l'unica "direzione" che funziona è essere long, non c'è + alpha di timing — c'è beta. Il trend portfolio (TP01) cattura quel beta in modo vol-targeted e + con DD ~12%, infinitamente meglio di qualunque regola di calendario qui. + +**Azione:** track F chiuso negativo. Non aggiungere nulla al portafoglio. Il soffitto Sharpe ~1.3 su +BTC/ETH regge. diff --git a/docs/diary/2026-06-19-trackG-prior-levels.md b/docs/diary/2026-06-19-trackG-prior-levels.md new file mode 100644 index 0000000..3a207bb --- /dev/null +++ b/docs/diary/2026-06-19-trackG-prior-levels.md @@ -0,0 +1,85 @@ +# Track G — Prior-period level breakouts / range (BTC & ETH, calendar-anchored) + +**Data:** 2026-06-19 · **Script:** `scripts/research/trackG_prior_levels.py` +**Harness:** `src/backtest/harness.py` (honest, entry decided at `close[i]`, fill `close[i]`). + +## Domanda + +Esistono edge net-positivi OOS, robusti su BTC **e** ETH, definiti rispetto a un **periodo +calendario precedente** (giorno/settimana/opening-range)? E soprattutto: i breakout di livello +**continuano** (trend) o **rientrano** (fade)? + +## No look-ahead (garanzie) + +- Livelli prior-day/week costruiti aggregando a barre giornaliere/settimanali (UTC) e poi + **`shift(1)`** sul frame del periodo *chiuso*: il periodo corrente vede solo il precedente + totalmente chiuso. Mai "oggi"/"questa settimana" nel livello. +- Opening-range usato **solo** sulle barre dopo la chiusura della finestra di apertura. +- Direzione + prezzo decisi a `close[i]`, fill a `close[i]`. Mai entry sul livello esatto intrabar. +- Bug iniziale corretto: mismatch tz-aware vs tz-naive nel mapping dei livelli (dava 0 trade). + +## Risultati (1h, fee 0.10% RT, leva 1x, OOS 65/35) + +### Continuation vs FADE — il verdetto è netto + +| Regola (PD = prior-day) | BTC OOS | ETH OOS | Sharpe OOS | +|---|---|---|---| +| **PD-high CONT (long su rottura max ieri)** | **+25%** | **+16%** | +0.5 / +0.3 | +| PD-high FADE | **−68%** | **−68%** | −1.6 / −1.2 | +| PD-low CONT (short su rottura min ieri) | −33% | −60% | −0.5 / −0.8 | +| PD-low FADE | −36% | −8% | −0.6 / +0.1 | + +- **I breakout CONTINUANO, non rientrano.** Il lato FADE è robustamente **negativo** su entrambi + gli asset (sia high che low), su prior-day, prior-week e opening-range. Conferma diretta della + tesi del reset: la mean-reversion / fade è morta su dati certificati. +- **Asimmetria long-only:** funziona solo la rottura del **massimo** (long), non quella del + **minimo** (short). Cioè non è un edge di breakout *simmetrico/direzione-neutro*: è cattura del + **drift/trend rialzista** del cripto. La PD-low-cont (short sui breakdown) perde perché in questo + campione il cripto sale. + +### Grid robustness (PASS 6) — survivor = OOS>0 su ENTRAMBI + +- **PD-high CONT: 3/3 celle** (buffer 0/0.1%/0.3%) positive OOS su BTC **e** ETH → robusto al buffer. +- PD-high fade, PD-low cont/fade, OR-fade: **0 survivor**. +- **OR-cont:** positiva solo su ETH, negativa su BTC su tutte le finestre (3/6/8/12h) → artefatto + mono-asset, scartato dalla regola "entrambi". + +### Anchor-hour sweep (PASS 5) — non è un'ora fortunata + +PD-high cont positiva su **21/24** ore UTC (BTC) e **20/24** (ETH). Non dipende da un singolo +anchor → coerente con un edge reale (ma vedi sotto: è beta di trend). + +### Fee sweep + per-anno (PD-high cont, full sample) + +``` +BTC RT%: 0.00→+571 0.05→+289 0.10→+126 0.15→ +31 0.20→ −24 (OOS: +84/+52/+25/+3/−15) +ETH RT%: 0.00→+1754 0.05→+1012 0.10→+567 0.15→+299 0.20→+139 (OOS: +67/+39/+16/−3/−19) +BTC per-anno: 2019 +39 2020 +104 2021 +7 2022 −42 2023 +24 2024 +27 2025 −16 2026 +3 +ETH per-anno: 2020 +164 2021 +160 2022 +7 2023 +1 2024 +12 2025 −4 2026 +7 +Sharpe full: BTC +0.48 (maxDD 55%, €/d 2k +0.88) · ETH +0.86 (maxDD 34%, €/d 2k +4.27) +``` + +- **Fee-fragile:** alla baseline 0.10% RT sopravvive (OOS +25/+16%), ma muore già a ~0.15-0.20% RT. + Margine di fee sottile (≈1.5x baseline e l'edge sparisce su OOS). ~1000-1100 trade in 8 anni. +- **Drawdown enormi** (BTC 55%) e anni negativi (2022 −42% BTC, 2025 −16%). + +## Verdetto + +- **Sì, esiste un edge net-positivo OOS su entrambi gli asset:** *PD-high continuation* (long + quando `close` supera il massimo di ieri, exit a fine giornata UTC). Robusto al buffer e + all'anchor-hour. **MA non è deployabile come miglioramento:** + 1. È **long-only drift capture**, non un breakout simmetrico (il lato short fallisce) → è una + versione **più debole e ridondante** del Trend Portfolio TP01 (Sharpe 0.48-0.86 vs 1.32). + 2. **Fee-fragile** (muore a ~1.5x la fee baseline) e con **drawdown** molto peggiori. +- **Il contributo scientifico vero è la conferma della direzione:** sui dati certificati i + breakout di livello-calendario **CONTINUANO**; il fade è morto (negativo robusto su PD/PW/OR, + entrambi gli asset). Nessuna sorpresa mean-reversion nascosta nei livelli giornalieri/settimanali. +- **Niente di nuovo da mettere in produzione.** TP01 resta la strategia vincente; i breakout + prior-period non aggiungono Sharpe (stessa beta di trend, peggio eseguita). + +## Come riprodurre + +```bash +uv run python scripts/research/trackG_prior_levels.py # full (1h + 15m, ~25s) +uv run python scripts/research/trackG_prior_levels.py --quick # 1h only +``` diff --git a/docs/diary/2026-06-19-trackH-volume-vol.md b/docs/diary/2026-06-19-trackH-volume-vol.md new file mode 100644 index 0000000..2556305 --- /dev/null +++ b/docs/diary/2026-06-19-trackH-volume-vol.md @@ -0,0 +1,71 @@ +# Track H — Volume, Range & Volatility-Regime signals (BTC/ETH, certified, >=12h) + +**Date:** 2026-06-19 +**Script:** `scripts/research/trackH_volume_vol.py` (runnable, self-contained) +**Question:** does any volume / range / volatility-regime signal ADD to the deployed winner +TP01 (vol-targeted trend portfolio, 12h, Sharpe ~1.32) — i.e. net-positive OOS on BOTH BTC & +ETH AND uncorrelated (|corr|<~0.3) — OR work as a regime filter that lifts TP01's Sharpe / cuts +its DD? + +## Method (honest) +- Same causal per-bar engine as `TrendPortfolio.net_returns`: build a continuous TARGET decided + with data `<= close[i]`, HOLD it during bar `i+1` (`pos_held[t]=target[t-1]`), gross = pos×ret, + fee on `|Δpos|`. Identical in spirit to `harness.backtest_signals` (decide≤close[i], fill at + close[i]); two discrete signals cross-checked through `backtest_signals` directly. +- All features (volume z-score, OBV, ranges, realized vol) use prior/rolling windows shifted so + bar `i` sees only `<= i`. 12h/1d resampled from certified 1h via `resample_tf` (label='left'), + consumed index-based with the +1 hold → no open-label leak. +- Fee 0.10% RT baseline + sweep 0.00–0.40% RT. OOS 65/35 + per-year. Grid on BOTH assets. + Turnover and correlation-to-TP01 reported for every signal. +- **>=12h only** (12h + 1d). Sub-12h excluded per the standing lesson (fees + HF-noise overfit + + the 4h open-label look-ahead trap). + +## Signals tested +VT-long (volatility-managed long), VolBreakout (volume-z-confirmed Donchian), OBV-trend, +VW-mom (volume-weighted momentum), RangeExpand (range-expansion breakout), NR7-break +(narrowest-range breakout), DeclVolRev (declining-volume fade/reversal). Plus regime overlays on +TP01: keep-low-vol, keep-high-vol, vol-managed ×1.5, OBV-up confirmation. + +## Results (12h headline, fee 0.10% RT) +| signal | corr→TP01 | OOS Sharpe BTC/ETH | note | +|---|---|---|---| +| VT-long | 0.66 / 0.69 | 0.80 / 0.14 | trend-in-disguise; weak OOS ETH | +| VolBreakout | 0.69 / 0.71 | 0.54 / 0.49 | profitable but correlated | +| OBV-trend | 0.61 / 0.63 | 0.96 / 0.68 | profitable but correlated; turnover ~75/yr | +| VW-mom | 0.64 / 0.67 | 0.98 / 0.74 | basically TSMOM; correlated | +| RangeExpand | 0.48 / 0.49 | 0.37 / 1.04 | lower corr but BTC weak; ETH negative on 1d | +| NR7-break | 0.48 / 0.49 | 0.79 / 0.02 | fails OOS on ETH | +| DeclVolRev | -0.15 / -0.11 | -1.15 / -0.44 | **negative even at zero fee** | + +Grid robustness (12h, % cells positive full+OOS on both assets): VW-mom 100%, VT-long 100%, +VolBreakout 96%, RangeExpand 96%, OBV-trend 75% — but the robust ones are precisely the ones +that are highly correlated to TP01. Fee sweep: trend-family signals survive to 0.40% RT; +DeclVolRev gets worse with fees (it trades constantly). + +## Regime filters on TP01 (12h, 50/50 portfolio) +| variant | full Sharpe | OOS Sharpe | maxDD | CAGR | turn/y | +|---|---|---|---|---|---| +| **TP01 baseline** | **1.32** | 0.90 | 13.3% | 16.2% | 11.5 | +| × keep LOW-vol | 0.94 | 1.11 | 14.1% | 7.7% | 9.5 | +| × keep HIGH-vol | 0.98 | 0.18 | 9.9% | 7.9% | 4.9 | +| × vol-managed ×1.5 | 1.33 | 0.96 | 17.9% | 18.1% | 15.4 | +| × OBV-up only | 1.49 | 1.04 | 10.1% | 14.4% | 18.2 | + +OBV-up filter across EMA span: full Sharpe 1.49–1.52 (span 15–30), DD 7–10%, but OOS gain is +marginal (0.90→1.04 at span 30) and fades for span≥45 (OOS 0.69–0.73). It cuts ~2pp CAGR and +raises turnover ~60%. + +## Verdict (honest) +- **No uncorrelated additive edge exists.** Every *profitable* volume/range/vol signal is trend + in disguise (corr 0.61–0.75 to TP01) → cannot raise the 50/50 portfolio Sharpe. The genuinely + lower-corr signals (RangeExpand, NR7 ~0.48) fail OOS on at least one asset. +- **Mean-reversion / declining-volume fade is dead** — negative net AND at zero fee on both + assets. Reconfirms the v2.0.0 contamination lesson; MR is not a real edge on certified data. +- **Vol-regime gating hurts** (keep-low / keep-high both drop Sharpe to ~0.95). The vol-managed + overlay is Sharpe-neutral but DD-worse. +- **The only non-harmful overlay is OBV-up trend-confirmation:** it cuts DD (13.3%→10.1%) and + nudges full Sharpe to ~1.49, but it is trend double-confirmation (de-risking), not new alpha; + it costs CAGR, raises turnover, and the OOS Sharpe gain is within noise and span-sensitive. It + is worth keeping in mind as a **defensive DD overlay**, not as a Sharpe improver. +- **Bottom line:** the ~1.3 portfolio-Sharpe ceiling on BTC/ETH-only **holds**. TP01 stays the + deployable winner. Volume/range/vol add nothing uncorrelated. diff --git a/docs/diary/2026-06-19-trackI-momentum-reversal.md b/docs/diary/2026-06-19-trackI-momentum-reversal.md new file mode 100644 index 0000000..887b5a5 --- /dev/null +++ b/docs/diary/2026-06-19-trackI-momentum-reversal.md @@ -0,0 +1,99 @@ +# Track I — Alternative momentum formulations + long-horizon reversal (2026-06-19) + +**Script:** `scripts/research/trackI_momentum_reversal.py` (self-contained, runnable). +**Universe:** BTC & ETH only. **TF:** 12h + 1d (sub-12h excluded by rule). **Harness:** identical +honest machinery to TP01 — direction decided `<= close[i]`, positions held next bar (`pos_held[1:] += tgt[:-1]`), vol-target by inverse PAST-ONLY realized vol (target 20%, lev cap 2x), NET fee 0.10% +RT on turnover, 50/50 BTC+ETH. OOS 65/35 + per-year + fee sweep (0.00–0.40% RT). Correlation to +TP01 net returns reported for every candidate. + +## Goal +(A) A momentum formulation that BEATS or DIVERSIFIES the canonical 1-3-6m sign-blend (TP01, +Sharpe ~1.32). (B) Does the classic LONG-HORIZON REVERSAL (fade ~12m winners) give an +uncorrelated positive overlay? + +## PART A — momentum formulations (12h, long-flat, vs TP01 Sharpe 1.32 / OOS 0.90 / DD 13.3%) + +| formulation | Sharpe | IS | **OOS** | CAGR | maxDD | corr→TP01 | BTC | ETH | +|---|---|---|---|---|---|---|---|---| +| baseline sign-blend 1-3-6m | 1.32 | 1.54 | 0.90 | +16% | 13.3% | 1.00 | 1.15 | 1.10 | +| (i) z-score cum-return (tanh) | **1.35** | 1.63 | 0.85 | +12% | **8.4%** | 0.96 | 1.30 | 1.00 | +| (ii) risk-adjusted momentum | 1.27 | 1.49 | 0.84 | +13% | 9.5% | 0.97 | 1.21 | 1.00 | +| (iii) EMA-cross trend | 0.81 | 0.91 | 0.62 | +11% | 25.1% | 0.85 | 0.89 | 0.53 | +| (iii-b) MACD (calendar spans) | **1.50** | **1.87** | 0.74 | +22% | 17.7% | 0.69 | 1.30 | 1.32 | +| (iv) Donchian breakout | 1.10 | 1.36 | 0.57 | +17% | 25.0% | 0.86 | 1.08 | 0.82 | +| (v) acceleration (Δ-momentum) | 1.28 | 1.82 | 0.35 | +14% | 14.2% | 0.66 | 1.25 | 0.81 | +| (vi) 12-1 skip momentum | 0.67 | 0.79 | 0.47 | +9% | 24.5% | 0.68 | 0.70 | 0.49 | + +Results are essentially identical at 1d. Read-out: + +- **Nothing cleanly beats the sign-blend OOS on both assets.** The headline-Sharpe leaders are + artefacts of in-sample fit: **MACD** posts IS 1.87 but OOS collapses to 0.74 (gap = overfit) with + a worse DD (17.7%); **acceleration** IS 1.82 → OOS **0.35** (worst OOS decay of all). Both fail. +- **(i) z-score continuous momentum** is the one mild, honest refinement: Sharpe 1.35 (≈baseline) + but **maxDD 8.4% vs 13.3%** — the continuous score scales down position when the cumulative move + is statistically small, de-risking the tails. OOS 0.85 (slightly below baseline 0.90), CAGR drops + 16%→12%. It's a smoother sibling of TP01, **not a new edge** (corr 0.96). +- (vi) 12-1 skip (classic equity "12-1" momentum) **does NOT help crypto**: skipping the recent + month removes the strongest part of the signal here → Sharpe 0.67, corr 0.68. Crypto momentum + lives in the recent window, opposite to the equity stylised fact. +- Breakout/Donchian and EMA-cross are strictly worse (high DD, weak OOS). + +## PART B — long-horizon reversal (fade past winners), 12h + +Long-short reversal (short ~12/18/24m winners, long losers, vol-targeted): + +| reversal LS | Sharpe | OOS | CAGR | maxDD | corr→TP01 | +|---|---|---|---|---|---| +| 12m | -0.77 | -1.15 | -14% | 73% | -0.51 | +| 18m | -0.36 | -0.75 | -8% | 58% | -0.47 | +| 24m | **+0.04** | -0.07 | -1% | 43% | **-0.32** | +| 12-18-24m | -0.46 | -0.72 | -8% | 57% | -0.54 | + +- **Long-horizon reversal is NOT a standalone edge.** Standalone it LOSES money (12m/18m strongly + negative; only 24m is ~flat at Sharpe 0.04, OOS −0.07, and even that fails "net-positive OOS on + both assets": BTC +0.10 / ETH −0.03). Fading crypto winners over a year just shorts the trend. +- It IS genuinely negatively correlated to TP01 (24m: corr −0.32; 12-18-24: −0.54), as expected + (it's the opposite sign of medium-term momentum). +- **Momentum + reversal blend** (long 1-6m momentum, brake on very-long extension): the variant + `mom(1-3-6) − 0.5·rev(12-24)` is the most interesting single-strategy result — Sharpe **1.38**, + **OOS 0.98** (> baseline 0.90), **maxDD 10.6%** (< 13.3%), both assets positive (BTC 1.25/ETH + 1.05), corr 0.91, fee-robust (1.43→1.22 across 0.00–0.40% RT). CAGR drops 16%→12%. It is TP01 + with a long-term-extension brake: a modest *risk-adjusted* improvement, not more return. + +## COMBINED — TP01 + best diversifier (blend net returns) + +TP01 alone: Sharpe 1.321, CAGR +16%, maxDD 13.3%, OOS 0.90. + +| combo | Sharpe | CAGR | maxDD | OOS | corr | +|---|---|---|---|---|---| +| TP01 + 20% reversal-24m (LS) | **1.411** | +13% | 11.5% | **1.06** | -0.32 | +| TP01 + 30% reversal-24m (LS) | 1.366 | +12% | 11.8% | 1.06 | -0.32 | +| TP01 + 20% reversal-12-18-24 (LS) | 1.350 | +11% | 10.6% | 0.84 | -0.54 | +| TP01 + 50% z-score | 1.348 | +14% | 9.5% | 0.89 | +0.96 | + +- Adding a small slice of **reversal-24m long-short** lifts portfolio Sharpe 1.32→1.41 and OOS + 0.90→1.06 while cutting DD to 11.5%. **But be skeptical:** the overlay is a ~zero-mean stream + (standalone Sharpe 0.04). The benefit is almost entirely **variance reduction from the negative + correlation, not added alpha** — and it COSTS return (CAGR 16%→13%). With a true-zero-edge + diversifier this Sharpe bump is fragile (it leans on the −0.32 correlation persisting OOS, and the + OOS sample is one 2022-24 crypto cycle). I would NOT deploy capital on a standalone-losing sleeve + to chase a 0.09 Sharpe point that is really de-risking. + +## Fee sweep (12h portfolio Sharpe) +baseline 1.37→1.18, z-score 1.38→1.24, MACD 1.52→1.45 (lowest turnover), blend 1.43→1.22, +reversal-24m 0.07→−0.02 (0.00→0.40% RT). All trend formulations survive realistic fees; reversal +has no positive margin to survive on. + +## VERDICT (honest) +- **Is there a momentum formulation that beats the 1-3-6m sign-blend? No — not OOS, not on both + assets.** MACD/acceleration look better in-sample but decay OOS (overfit + higher DD). The only + honest refinement is **continuous z-score momentum**, which matches the Sharpe with materially + lower drawdown (8.4% vs 13.3%) — a smoother variant of the SAME edge, not a new one (corr 0.96). +- **Does long-horizon reversal give an uncorrelated positive overlay? No, not a real one.** It is + uncorrelated/negatively-correlated (good) but **not positive** standalone (it loses, or at best is + flat at 24m and fails the both-assets bar). The combined-Sharpe lift (→1.41) is variance reduction + from a near-zero-mean stream and sacrifices CAGR — fragile, not bankable alpha. +- **The ~1.3 structural Sharpe ceiling on BTC/ETH-only holds.** TP01 remains the deployable winner. + If anything, swap the sign-blend for the **z-score continuous score** (or the `mom − 0.5·rev` + brake) for a lower-DD profile at equal Sharpe — a risk-management tweak, not a return upgrade. diff --git a/scripts/research/options_real_quote_check.py b/scripts/research/options_real_quote_check.py new file mode 100644 index 0000000..2540bd7 --- /dev/null +++ b/scripts/research/options_real_quote_check.py @@ -0,0 +1,187 @@ +"""VERIFICA SLEEVE OPZIONI su QUOTE REALI Deribit — quanto Sharpe sopravvive a bid/ask + skew. + +Lo sleeve income della strategia esterna `crypto_backtest` (vendita di put settimanali CSP su +BTC, delta 0.28) e' backtestato su prezzi MODELLATI: Black-Scholes prezzato con DVOL = IV ATM, e +si incassa il premio "fair" (mid). Due gap reali NON catturati: + (1) BID/ASK: vendendo si incassa il BID, non il mid. + (2) SKEW: una put OTM (delta 0.28) ha IV piu' alta della ATM (DVOL) -> il modello prezza la put + con la vol sbagliata. + +Questo script: + PARTE 1 (rete, Deribit mainnet pubblico): scarica la catena REALE della scadenza ~settimanale, + trova la put a delta ~0.28, e misura: + - premio reale incassabile (BID, in USD) vs premio modellato (BS @ IV ATM) + - skew: IV della put OTM (mark) vs IV ATM (mark) + - spread: bid/mark + - HAIRCUT netto f = premio_bid_reale / premio_BS@ATM + PARTE 2 (locale): ri-esegue lo sleeve CSP settimanale (dati + modulo del progetto esterno) con + il premio moltiplicato per f -> Sharpe/CAGR/maxDD reali stimati, vs i modellati. + +NB ONESTO: e' UNO SNAPSHOT (la catena di oggi). Lo spread si allarga nello stress; lo skew varia. +Va ripetuto nel tempo per robustezza. Ma misura direttamente i due gap col mercato vero. + + uv run python scripts/research/options_real_quote_check.py +""" +from __future__ import annotations + +import sys +from pathlib import Path + +import numpy as np +import pandas as pd + +EXT = Path("/home/adriano/crypto_backtest") +sys.path.insert(0, str(EXT)) + +PUT_DELTA = 0.28 +CYCLE_DAYS = 7 +ANN = 365 + + +def fetch_real_chain(): + import ccxt + ex = ccxt.deribit({"enableRateLimit": True}) + ex.load_markets() + puts = [m for m in ex.markets.values() + if m.get("option") and m["base"] == "BTC" and m["optionType"] == "put"] + calls = [m for m in ex.markets.values() + if m.get("option") and m["base"] == "BTC" and m["optionType"] == "call"] + # expiries -> pick the one closest to CYCLE_DAYS days out + now = pd.Timestamp.utcnow().tz_localize(None) + def exp_dt(m): + return pd.to_datetime(m["symbol"].split("-")[1], format="%y%m%d") + exps = sorted(set(exp_dt(m) for m in puts)) + target = now + pd.Timedelta(days=CYCLE_DAYS) + expiry = min(exps, key=lambda e: abs((e - target).days)) + dte = (expiry - now).days + (expiry - now).seconds / 86400 + chain_puts = [m for m in puts if exp_dt(m) == expiry] + chain_calls = [m for m in calls if exp_dt(m) == expiry] + print(f" scadenza scelta: {expiry.date()} (DTE ~{dte:.1f}g, target {CYCLE_DAYS}g) " + f"strikes put: {len(chain_puts)}") + + def tick(m): + try: + t = ex.fetch_ticker(m["symbol"]) + i = t["info"] + g = i.get("greeks") or {} + return dict(symbol=m["symbol"], strike=float(m["strike"]), + delta=float(g.get("delta", "nan")), mark_iv=float(i.get("mark_iv", "nan")), + bid=float(i.get("best_bid_price") or 0), ask=float(i.get("best_ask_price") or 0), + mark=float(i.get("mark_price") or 0), + S=float(i.get("underlying_price") or i.get("index_price") or 0)) + except Exception: + return None + + rows = [r for r in (tick(m) for m in chain_puts) if r and np.isfinite(r["delta"])] + callrows = [r for r in (tick(m) for m in chain_calls) if r and np.isfinite(r["delta"])] + return expiry, dte, pd.DataFrame(rows), pd.DataFrame(callrows) + + +def bs_put(S, K, T, sigma): + from scipy.stats import norm + if T <= 0 or sigma <= 0: + return max(0.0, K - S) + d1 = (np.log(S / K) + 0.5 * sigma ** 2 * T) / (sigma * np.sqrt(T)) + d2 = d1 - sigma * np.sqrt(T) + return K * norm.cdf(-d2) - S * norm.cdf(-d1) + + +def measure_haircut(dte, puts, calls): + S = puts["S"].iloc[0] + T = dte / ANN + # ATM IV: option with |delta| closest to 0.5 (use calls+puts mark_iv near ATM) + allo = pd.concat([puts.assign(typ="P"), calls.assign(typ="C")], ignore_index=True) + atm = allo.iloc[(allo["delta"].abs() - 0.5).abs().argsort()[:4]] + atm_iv = atm["mark_iv"].mean() / 100.0 + # delta-0.28 put (delta negative) + p = puts.iloc[(puts["delta"] - (-PUT_DELTA)).abs().argsort()[:1]].iloc[0] + K = p["strike"] + put_iv = p["mark_iv"] / 100.0 + # premiums in USD (Deribit option price is in BTC) + bid_usd = p["bid"] * S + mark_usd = p["mark"] * S + ask_usd = p["ask"] * S + bs_atm_usd = bs_put(S, K, T, atm_iv) # cio' che il backtest assume (DVOL=ATM, incassa mid) + bs_skew_usd = bs_put(S, K, T, put_iv) # BS alla vol REALE della put (isola lo skew) + + print("\n --- MISURA SU QUOTE REALI (snapshot) ---") + print(f" underlying S = {S:,.0f} strike(delta~-0.28) K = {K:,.0f} ({(1-K/S)*100:.1f}% OTM) delta {p['delta']:.3f}") + print(f" IV ATM (DVOL-equivalente) = {atm_iv*100:.1f}% IV put OTM (mark) = {put_iv*100:.1f}% " + f"skew +{(put_iv-atm_iv)*100:.1f} pt") + print(f" premio put (USD): BID {bid_usd:,.1f} mark {mark_usd:,.1f} ask {ask_usd:,.1f}") + print(f" spread bid/mark = {(p['bid']/p['mark']) if p['mark']>0 else float('nan'):.3f} " + f"(ask-bid)/mark = {((p['ask']-p['bid'])/p['mark']) if p['mark']>0 else float('nan'):.3f}") + print(f" modellato dal backtest BS@IV-ATM = {bs_atm_usd:,.1f} USD (BS@IV-put-reale = {bs_skew_usd:,.1f})") + f_bid = bid_usd / bs_atm_usd if bs_atm_usd > 0 else float("nan") + f_mark = mark_usd / bs_atm_usd if bs_atm_usd > 0 else float("nan") + print(f" HAIRCUT premio: reale(BID)/modello = {f_bid:.3f} | mark/modello = {f_mark:.3f}") + print(f" -> lo skew ALZA il premio lordo (+{(bs_skew_usd/bs_atm_usd-1)*100:.0f}% vs ATM), ma il " + f"BID/ask lo riporta a {f_bid*100:.0f}% del modello.") + return f_bid + + +def csp_sleeve_haircut(f): + """Ri-esegue lo sleeve CSP settimanale (dati+modulo esterni) con premio * f.""" + import options_deribit as od + px = pd.read_csv(EXT / "data/BTCUSDT.csv", parse_dates=["date"]).set_index("date")["close"] + dvol = pd.read_csv(EXT / "data/DVOL_BTC.csv", parse_dates=["date"]).set_index("date")["close"] + iv = od.build_iv(px, "BTC", dvol) + d0 = dvol.index[0] + px, iv = px[px.index >= d0], iv[iv.index >= d0] + + def sim(prem_mult, m=0.63): + idx = px.index + locs = list(range(0, len(idx) - CYCLE_DAYS, CYCLE_DAYS)) + T = CYCLE_DAYS / ANN + rows = [] + for i in locs: + S0, S1, sig = px.iloc[i], px.iloc[i + CYCLE_DAYS], iv.iloc[i] + if not (np.isfinite(S0) and np.isfinite(S1) and np.isfinite(sig)): + continue + Kp = od.strike_for_delta(S0, T, sig, PUT_DELTA, call=False) + pp = od.bs_price(S0, Kp, T, sig, call=False) * prem_mult # <-- haircut sul premio + fee = od.option_fee(S0, pp) + (od.SETTLE_FEE * S0 if S1 < Kp else 0) + pnl = pp - max(Kp - S1, 0.0) - fee + rows.append((idx[i + CYCLE_DAYS], m * pnl / S0)) + s = pd.Series({d: r for d, r in rows}).sort_index() + return s + + def met(s, name): + eq = (1 + s).cumprod() + cpy = ANN / CYCLE_DAYS + yrs = len(s) / cpy + cagr = eq.iloc[-1] ** (1 / yrs) - 1 if eq.iloc[-1] > 0 else -1 + sh = s.mean() / s.std() * np.sqrt(cpy) + dd = (eq / eq.cummax() - 1).min() + print(f" {name:<34s} CAGR {cagr*100:>+6.1f}% Sharpe {sh:>5.2f} maxDD {dd*100:>6.1f}% win {(s>0).mean()*100:>3.0f}%") + return sh + + print("\n --- RI-ESECUZIONE SLEEVE CSP con HAIRCUT REALE (m=0.63, hold-to-expiry) ---") + print(f" finestra {px.index[0].date()} -> {px.index[-1].date()} (DVOL reale)") + sh_model = met(sim(1.00), "modello (premio pieno, BS@DVOL)") + sh_real = met(sim(f), f"reale stimato (premio x{f:.2f} = BID)") + # sensitivity + for ff in (0.85, 0.70, 0.55): + met(sim(ff), f"sensitivity premio x{ff:.2f}") + print(f"\n => con haircut reale f={f:.2f}: Sharpe sleeve {sh_model:.2f} -> {sh_real:.2f}") + return sh_model, sh_real + + +def main(): + print("=" * 92) + print("# VERIFICA SLEEVE OPZIONI su QUOTE REALI DERIBIT — quanto Sharpe sopravvive") + print("=" * 92) + try: + expiry, dte, puts, calls = fetch_real_chain() + f = measure_haircut(dte, puts, calls) + except Exception as e: + print(f" [rete] impossibile scaricare la catena reale ({type(e).__name__}: {e})") + print(" uso haircut di letteratura f=0.70 (spread+skew tipici su put OTM settimanali)") + f = 0.70 + f = float(np.clip(f, 0.3, 1.2)) + csp_sleeve_haircut(f) + print("\n CAVEAT: snapshot singolo; spread peggiora nello stress; ripetere nel tempo + testnet.") + + +if __name__ == "__main__": + main() diff --git a/scripts/research/trackD_lookahead_audit.py b/scripts/research/trackD_lookahead_audit.py new file mode 100644 index 0000000..e788a2b --- /dev/null +++ b/scripts/research/trackD_lookahead_audit.py @@ -0,0 +1,118 @@ +"""ADVERSARIAL LOOK-AHEAD / EXECUTION-LAG AUDIT of the trend portfolio across timeframes. + +Motivation (2026-06-19): a look-ahead bug (ffill mixed-timeframe on open-labeled bars) can +inflate sub-daily Sharpe (e.g. 4h to ~1.6 vs a real ~1.1). This audit stress-tests OUR pipeline: + + 1. EXECUTION LAG: standard book holds the position decided at close[i] during bar i+1. + We re-run with an EXTRA bar of delay (held during i+2) — i.e. you cannot trade exactly at + the close; there is one bar of slippage/latency. A genuine slow-trend edge barely moves; a + timing artifact collapses. We sweep lag = 1 (standard) and 2 (conservative). + 2. RELABEL TEST: resample with label='left' (open-labeled, our default) vs label='right' + (close-labeled). The realized Sharpe must be (near) identical; a large gap => the labeling + leaks information. + +Conclusion target: identify the timeframe below which costs + lag dominate (don't deploy there). + +Run: uv run python scripts/research/trackD_lookahead_audit.py +""" +from __future__ import annotations + +import sys +from pathlib import Path + +import numpy as np +import pandas as pd + +sys.path.insert(0, str(Path(__file__).resolve().parents[2])) + +from src.backtest.harness import load +from src.strategies.trend_portfolio import simple_returns, realized_vol, tsmom_blend + +ASSETS = ["BTC", "ETH"] +FEE_SIDE = 0.0005 +TARGET_VOL = 0.20 +LEVERAGE = 2.0 +LONG_ONLY = True +TFS = {"4h": ("4h", 6), "6h": ("6h", 4), "8h": ("8h", 3), "12h": ("12h", 2), "1d": ("1D", 1)} + + +def resample(df1h: pd.DataFrame, rule: str, label: str) -> pd.DataFrame: + g = df1h.copy() + idx = pd.to_datetime(g["timestamp"], unit="ms", utc=True) + idx.name = "dt" + g.index = idx + out = g.resample(rule, label=label, closed="left").agg( + {"open": "first", "high": "max", "low": "min", "close": "last", "volume": "sum"}) + out = out.dropna(subset=["open"]) + out["datetime"] = out.index + return out.reset_index(drop=True) + + +def target_series(c, bpd): + bpy = bpd * 365.25 + r = simple_returns(c) + vol = realized_vol(r, 30 * bpd, bpy) + direction = np.clip(tsmom_blend(c, (30 * bpd, 90 * bpd, 180 * bpd)), 0, None) if LONG_ONLY \ + else tsmom_blend(c, (30 * bpd, 90 * bpd, 180 * bpd)) + scal = np.where((vol > 0) & np.isfinite(vol), TARGET_VOL / vol, 0.0) + tgt = np.clip(direction * scal, -LEVERAGE, LEVERAGE) + tgt[~np.isfinite(tgt)] = 0.0 + return tgt, r + + +def sleeve_net(df, bpd, lag): + """net[t] uses position decided at close[t-lag] (lag>=1). lag=1 = standard, lag=2 = +1 delay.""" + c = df["close"].values.astype(float) + tgt, r = target_series(c, bpd) + pos = np.zeros(len(tgt)) + pos[lag:] = tgt[:-lag] + gross = pos * r + turn = np.abs(np.diff(pos, prepend=0.0)) + net = gross - FEE_SIDE * turn + net[:lag] = 0.0 + return np.clip(net, -0.99, None), pd.to_datetime(df["datetime"]) + + +def portfolio_metrics(dfs, bpd, lag): + series = {} + for a in ASSETS: + net, ts = sleeve_net(dfs[a], bpd, lag) + series[a] = pd.Series(net, index=pd.to_datetime(ts.values)) + J = pd.concat(series, axis=1, join="inner").dropna() + combo = 0.5 * J["BTC"].values + 0.5 * J["ETH"].values + bpy = bpd * 365.25 + sh = float(np.mean(combo) / np.std(combo) * np.sqrt(bpy)) if np.std(combo) > 0 else 0.0 + eq = np.cumprod(1.0 + np.clip(combo, -0.99, None)) + dd = float(np.max((np.maximum.accumulate(eq) - eq) / np.maximum.accumulate(eq))) + yrs = (J.index[-1] - J.index[0]).days / 365.25 + cagr = eq[-1] ** (1 / yrs) - 1 + return sh, dd, cagr + + +def main(): + raw = {a: load(a, "1h") for a in ASSETS} + print("=" * 96) + print("# LOOK-AHEAD / EXECUTION-LAG AUDIT — trend portfolio (long-flat, tvol20, lev2), per timeframe") + print("# lag1 = standard (decision held next bar). lag2 = +1 bar execution delay (conservative).") + print("# left/right = resample label (open vs close). Big gap => labeling leak.") + print("=" * 96) + print(f" {'TF':<5s}{'Sh lag1(L)':>12s}{'Sh lag2(L)':>12s}{'Sh lag1(R)':>12s}" + f"{'CAGR l1':>10s}{'CAGR l2':>10s}{'DD l1':>8s}{'lag-decay':>11s}") + for tf, (rule, bpd) in TFS.items(): + dfsL = {a: resample(raw[a], rule, "left") for a in ASSETS} + dfsR = {a: resample(raw[a], rule, "right") for a in ASSETS} + sh1L, dd1, cagr1 = portfolio_metrics(dfsL, bpd, 1) + sh2L, _, cagr2 = portfolio_metrics(dfsL, bpd, 2) + sh1R, _, _ = portfolio_metrics(dfsR, bpd, 1) + decay = (sh1L - sh2L) / sh1L * 100 if sh1L else 0.0 + flag = " <-- robust" if sh2L >= 0.9 * sh1L and abs(sh1L - sh1R) < 0.1 else "" + print(f" {tf:<5s}{sh1L:>12.2f}{sh2L:>12.2f}{sh1R:>12.2f}" + f"{cagr1*100:>+9.1f}%{cagr2*100:>+9.1f}%{dd1*100:>7.1f}%{decay:>+10.0f}%{flag}") + print("\n Interpretation:") + print(" - If Sh lag2 << Sh lag1 (big lag-decay), the edge needs to trade AT the close -> sub-TF") + print(" timing artifact / cost-fragile. Robust slow-trend should barely move with +1 bar.") + print(" - If Sh lag1(left) != Sh lag1(right), the bar LABELING leaks -> look-ahead. Should match.") + + +if __name__ == "__main__": + main() diff --git a/scripts/research/trackF_seasonality.py b/scripts/research/trackF_seasonality.py new file mode 100644 index 0000000..26dba86 --- /dev/null +++ b/scripts/research/trackF_seasonality.py @@ -0,0 +1,365 @@ +"""TRACK F — CALENDAR SEASONALITY on BTC & ETH (hour-of-day, day-of-week, interactions). + +Honest test of whether there is a SYSTEMATIC, TRADEABLE calendar edge on the certified +Deribit-mainnet BTC/ETH feeds. Seasonality is the easiest place on earth to overfit +(24 hours x 7 weekdays = 168 buckets => you WILL find "significant" cells by chance), so +every claim here is held to the project's anti-look-ahead, OOS, per-year, both-assets bar. + +METHODOLOGY (no shortcuts): + - ret[i] = close[i]/close[i-1]-1 is known at close[i]. A position decided at close[i] + earns ret[i+1]. We NEVER include the bar being traded (or any future bar) in the + statistic that decides the trade. + - DESCRIPTIVE tables (per-hour / per-weekday mean returns) are split IS(65%)/OOS(35%). + They are diagnostics, not trades. + - TRADEABLE rule = ADAPTIVE EXPANDING sign: at close[i] we look up the calendar bucket + of bar i+1 (the clock is known with zero look-ahead) and take the SIGN of that bucket's + mean return computed ONLY on bars <= i (expanding, warmup-gated). Long-flat or + long-short. Fees charged only on |Δposition| (turnover-aware). This lets the data pick + each bucket's sign LIVE — the honest analogue of "trade the seasonal". + - Also an in-sample-optimised discrete rule (enter at hour H, hold W bars, best dir) is + shown ONLY to demonstrate the overfit gap IS->OOS. + - NET fees fee_side baseline 0.0005 (=0.10% RT); swept 0.0005/0.00075/0.001. + - A survivor must be net-positive OOS AND across years AND on BOTH BTC & ETH. + +Run: uv run python scripts/research/trackF_seasonality.py +""" +from __future__ import annotations + +import sys +from pathlib import Path + +import numpy as np +import pandas as pd + +sys.path.insert(0, str(Path(__file__).resolve().parents[2])) + +from src.backtest.harness import load # noqa: E402 + +ASSETS = ["BTC", "ETH"] +TF = "1h" +FEE_SIDE = 0.0005 # 0.05%/side = 0.10% round-trip +BARS_PER_DAY = 24 +BPY = BARS_PER_DAY * 365.25 + + +# --------------------------------------------------------------------------- +# helpers +# --------------------------------------------------------------------------- +def prep(asset: str, tf: str = TF): + df = load(asset, tf) + c = df["close"].values.astype(float) + ret = np.empty(len(c)) + ret[0] = 0.0 + ret[1:] = c[1:] / c[:-1] - 1.0 + dt = pd.to_datetime(df["datetime"]) + return dict( + df=df, ret=ret, + hour=dt.dt.hour.values.astype(int), + dow=dt.dt.dayofweek.values.astype(int), # 0=Mon..6=Sun + ts=dt, + ) + + +def metrics_from_pnl(pnl: np.ndarray, ts: pd.Series): + """pnl[i] = realized per-bar net return of the strategy (already fee-adjusted).""" + eq = np.cumprod(1.0 + np.clip(pnl, -0.99, None)) + r = pnl[np.isfinite(pnl)] + sharpe = float(np.mean(r) / np.std(r) * np.sqrt(BPY)) if np.std(r) > 0 else 0.0 + peak = np.maximum.accumulate(eq) + maxdd = float(np.max((peak - eq) / peak)) if len(eq) else 0.0 + span_days = (ts.iloc[-1] - ts.iloc[0]).total_seconds() / 86400 + years = span_days / 365.25 if span_days > 0 else 1.0 + total = eq[-1] / eq[0] if len(eq) else 1.0 + cagr = total ** (1 / years) - 1 if years > 0 and total > 0 else -1.0 + daily_2k = (2000 * total - 2000) / span_days if span_days > 0 else 0.0 + return dict(sharpe=sharpe, maxdd=maxdd, cagr=cagr, total=total - 1.0, + daily_2k=daily_2k, eq=eq) + + +def per_year_pnl(pnl: np.ndarray, ts: pd.Series): + s = pd.Series(pnl, index=ts.values) + out = {} + for y, g in s.groupby(s.index.year): + eq = np.cumprod(1.0 + np.clip(g.values, -0.99, None)) + out[int(y)] = float(eq[-1] - 1.0) + return out + + +# --------------------------------------------------------------------------- +# 1. DESCRIPTIVE seasonality tables (diagnostics, IS vs OOS) +# --------------------------------------------------------------------------- +def descriptive(data, frac=0.65): + n = len(data["ret"]) + cut = int(n * frac) + ret, hour, dow = data["ret"], data["hour"], data["dow"] + rows_h, rows_d = {}, {} + for h in range(24): + m_is = ret[:cut][hour[:cut] == h] + m_oos = ret[cut:][hour[cut:] == h] + rows_h[h] = (m_is.mean() * 1e4, m_oos.mean() * 1e4, + np.sign(m_is.mean()) == np.sign(m_oos.mean())) + for d in range(7): + m_is = ret[:cut][dow[:cut] == d] + m_oos = ret[cut:][dow[cut:] == d] + rows_d[d] = (m_is.mean() * 1e4, m_oos.mean() * 1e4, + np.sign(m_is.mean()) == np.sign(m_oos.mean())) + return rows_h, rows_d + + +# --------------------------------------------------------------------------- +# 2. ADAPTIVE EXPANDING-sign seasonal strategy (the honest tradeable test) +# --------------------------------------------------------------------------- +def adaptive_seasonal(data, bucket="hour", mode="longshort", + warmup=200, fee_side=FEE_SIDE): + """Position at close[i] = sign of the EXPANDING past mean return of bar (i+1)'s + calendar bucket, using only bars <= i. earns ret[i+1]. Fee on |Δposition|.""" + ret = data["ret"] + key = data[bucket] + n = len(ret) + nbuck = int(key.max()) + 1 + sums = np.zeros(nbuck) + counts = np.zeros(nbuck) + pos = np.zeros(n) + for i in range(1, n - 1): + b = key[i] + sums[b] += ret[i] + counts[b] += 1 + nb = key[i + 1] + if counts[nb] >= warmup: + m = sums[nb] / counts[nb] + if m > 0: + pos[i] = 1.0 + else: + pos[i] = -1.0 if mode == "longshort" else 0.0 + # pnl[i] earned over bar i+1 + pnl = np.zeros(n) + prev = 0.0 + for i in range(1, n - 1): + turn = abs(pos[i] - prev) + pnl[i] = pos[i] * ret[i + 1] - fee_side * turn + prev = pos[i] + return pnl, pos + + +def adaptive_hourxdow(data, mode="longshort", warmup=120, fee_side=FEE_SIDE): + ret, hour, dow = data["ret"], data["hour"], data["dow"] + key = hour * 7 + dow # 168 buckets + n = len(ret) + sums = np.zeros(168) + counts = np.zeros(168) + pos = np.zeros(n) + for i in range(1, n - 1): + b = key[i] + sums[b] += ret[i] + counts[b] += 1 + nb = key[i + 1] + if counts[nb] >= warmup: + m = sums[nb] / counts[nb] + if m > 0: + pos[i] = 1.0 + else: + pos[i] = -1.0 if mode == "longshort" else 0.0 + pnl = np.zeros(n) + prev = 0.0 + for i in range(1, n - 1): + turn = abs(pos[i] - prev) + pnl[i] = pos[i] * ret[i + 1] - fee_side * turn + prev = pos[i] + return pnl, pos + + +# --------------------------------------------------------------------------- +# 3. In-sample-optimised DISCRETE rule (to expose the overfit gap) +# --------------------------------------------------------------------------- +def discrete_hour_rule_scan(data, frac=0.65, fee_side=FEE_SIDE): + """Scan IS for best (entry_hour, hold_window, direction) by IS Sharpe; report OOS. + + A trade: enter at close of bar whose hour==H (decided with data<=close[i]), hold W + bars, exit at close. One trade per day. Fee charged round-trip on each trade. + """ + ret, hour, ts = data["ret"], data["hour"], data["ts"] + n = len(ret) + cut = int(n * frac) + + def rule_pnl(H, W, direction, lo, hi): + pnl = np.zeros(n) + i = lo + last_exit = lo - 1 + while i < hi: + if hour[i] == H and i > last_exit: + # cumulative return over the next W bars: prod(1+ret[i+1..i+W]) - 1 + end = min(i + W, n - 1) + gross = np.prod(1.0 + ret[i + 1:end + 1]) - 1.0 + pnl[i] = direction * gross - 2 * fee_side + last_exit = end + i = end + else: + i += 1 + return pnl + + best = None + n_tested = 0 + for H in range(24): + for W in (1, 2, 3, 4, 6, 8, 12, 24): + for direction in (+1, -1): + n_tested += 1 + pnl_is = rule_pnl(H, W, direction, 1, cut) + r = pnl_is[pnl_is != 0.0] + if len(r) < 50: + continue + sh = np.mean(r) / np.std(r) * np.sqrt(BPY) if np.std(r) > 0 else 0.0 + if best is None or sh > best[0]: + best = (sh, H, W, direction) + sh, H, W, direction = best + pnl_oos = rule_pnl(H, W, direction, cut, n) + r_oos = pnl_oos[pnl_oos != 0.0] + sh_oos = (np.mean(r_oos) / np.std(r_oos) * np.sqrt(BPY)) if (len(r_oos) and np.std(r_oos) > 0) else 0.0 + return dict(n_tested=n_tested, H=H, W=W, dir=direction, sh_is=sh, + sh_oos=sh_oos, n_is=int((rule_pnl(H, W, direction, 1, cut) != 0).sum()), + n_oos=len(r_oos), oos_mean_bp=r_oos.mean() * 1e4 if len(r_oos) else 0.0) + + +# --------------------------------------------------------------------------- +# reporting +# --------------------------------------------------------------------------- +def split_metrics(pnl, ts, frac=0.65): + n = len(pnl) + cut = int(n * frac) + m_is = metrics_from_pnl(pnl[:cut], ts.iloc[:cut]) + m_oos = metrics_from_pnl(pnl[cut:], ts.iloc[cut:]) + m_all = metrics_from_pnl(pnl, ts) + return m_is, m_oos, m_all + + +def turnover_per_year(pos, ts): + s = pd.Series(np.abs(np.diff(pos, prepend=0.0)), index=ts.values) + return s.groupby(s.index.year).sum().to_dict() + + +def main(): + print("=" * 100) + print("# TRACK F — CALENDAR SEASONALITY (hour-of-day / day-of-week / hour×weekday)") + print("# certified Deribit-mainnet BTC & ETH, 1h UTC. fee_side=0.0005 (0.10% RT).") + print("# No look-ahead: bucket stats use only bars <= i; position earns ret[i+1].") + print("=" * 100) + + data = {a: prep(a) for a in ASSETS} + + # --- DESCRIPTIVE --------------------------------------------------------- + print("\n" + "#" * 100) + print("# 1. DESCRIPTIVE per-bucket mean returns (basis points/bar). IS=first 65%, OOS=last 35%.") + print("# 'sign?' = IS and OOS agree on sign. Diagnostics only (NOT trades, no fees).") + print("#" * 100) + for a in ASSETS: + rows_h, rows_d = descriptive(data[a]) + print(f"\n ── {a} HOUR-OF-DAY (UTC) mean bp/hr ─────────────────────────────") + print(" hr : IS_bp OOS_bp sign?") + agree_h = 0 + for h in range(24): + iv, ov, ag = rows_h[h] + agree_h += int(ag) + flag = " <-- US open" if h in (13, 14) else (" <-- US close" if h in (20, 21) else "") + print(f" {h:>2d} : {iv:>+6.2f} {ov:>+6.2f} {'Y' if ag else '.'}{flag}") + print(f" hour sign-agreement IS/OOS: {agree_h}/24") + print(f"\n ── {a} DAY-OF-WEEK mean bp/bar (0=Mon..6=Sun) ──────────────────") + names = ["Mon", "Tue", "Wed", "Thu", "Fri", "Sat", "Sun"] + agree_d = 0 + for d in range(7): + iv, ov, ag = rows_d[d] + agree_d += int(ag) + print(f" {names[d]} : {iv:>+6.3f} {ov:>+6.3f} {'Y' if ag else '.'}") + print(f" weekday sign-agreement IS/OOS: {agree_d}/7") + + # --- ADAPTIVE EXPANDING-SIGN (the honest tradeable test) ---------------- + print("\n" + "#" * 100) + print("# 2. ADAPTIVE EXPANDING-SIGN seasonal strategies (HONEST tradeable test).") + print("# sign of bucket's PAST-ONLY mean decides position; fee on turnover.") + print("#" * 100) + configs = [ + ("HOUR long-short", "hour", "longshort", 200), + ("HOUR long-flat ", "hour", "longflat", 200), + ("DOW long-short", "dow", "longshort", 60), + ("DOW long-flat ", "dow", "longflat", 60), + ] + for label, bucket, mode, warmup in configs: + print(f"\n ── {label} ────────────────────────────────────────────────────") + for a in ASSETS: + pnl, pos = adaptive_seasonal(data[a], bucket=bucket, mode=mode, warmup=warmup) + ts = data[a]["ts"] + m_is, m_oos, m_all = split_metrics(pnl, ts) + py = per_year_pnl(pnl, ts) + yrs = "".join(f"{py.get(y, float('nan'))*100:>+6.0f}" for y in range(2019, 2027)) + print(f" {a}: ALL Sh={m_all['sharpe']:>+5.2f} CAGR={m_all['cagr']*100:>+6.1f}% " + f"DD={m_all['maxdd']*100:>4.1f}% €/d={m_all['daily_2k']:>+5.2f} | " + f"IS Sh={m_is['sharpe']:>+5.2f} OOS Sh={m_oos['sharpe']:>+5.2f}") + print(f" per-year %: {yrs} (2019..2026)") + + # buy-and-hold benchmark — the key control: does any 'seasonal' beat just being long? + print(f"\n ── BUY-AND-HOLD benchmark (the control for long-bias) ──") + for a in ASSETS: + ret = data[a]["ret"].copy() + ret[0] = 0.0 + m = metrics_from_pnl(ret, data[a]["ts"]) + print(f" {a}: Sh={m['sharpe']:>+5.2f} CAGR={m['cagr']*100:>+6.1f}% DD={m['maxdd']*100:>4.1f}% " + f" <- compare to DOW long-flat above (it's nearly identical = no edge, just long)") + + # hour x weekday interaction (168 buckets — extreme overfit risk) + print(f"\n ── HOUR×WEEKDAY long-short (168 buckets, warmup 120) — overfit canary ──") + for a in ASSETS: + pnl, pos = adaptive_hourxdow(data[a], mode="longshort", warmup=120) + ts = data[a]["ts"] + m_is, m_oos, m_all = split_metrics(pnl, ts) + print(f" {a}: ALL Sh={m_all['sharpe']:>+5.2f} CAGR={m_all['cagr']*100:>+6.1f}% " + f"DD={m_all['maxdd']*100:>4.1f}% | IS Sh={m_is['sharpe']:>+5.2f} OOS Sh={m_oos['sharpe']:>+5.2f}") + + # --- FEE SWEEP on the best adaptive config ------------------------------- + print("\n" + "#" * 100) + print("# 3. FEE SWEEP — HOUR long-short adaptive (turnover-aware). Are survivors fee-robust?") + print("#" * 100) + for fee in (0.0, 0.0005, 0.00075, 0.001): + line = f" fee_side={fee:.5f} (RT {fee*2*100:.2f}%): " + for a in ASSETS: + pnl, _ = adaptive_seasonal(data[a], bucket="hour", mode="longshort", + warmup=200, fee_side=fee) + m = metrics_from_pnl(pnl, data[a]["ts"]) + line += f"{a} Sh={m['sharpe']:>+5.2f} CAGR={m['cagr']*100:>+6.1f}% " + print(line) + + # --- TURNOVER (fees are first-order for hour strategies) ----------------- + print("\n" + "#" * 100) + print("# 4. TURNOVER (HOUR long-short adaptive): position flips/year (each flip costs ~fee).") + print("#" * 100) + for a in ASSETS: + _, pos = adaptive_seasonal(data[a], bucket="hour", mode="longshort", warmup=200) + tpy = turnover_per_year(pos, data[a]["ts"]) + s = " ".join(f"{y}:{int(v)}" for y, v in sorted(tpy.items())) + print(f" {a} turnover units/yr: {s}") + + # --- IN-SAMPLE-OPTIMISED DISCRETE RULE (overfit demonstration) ---------- + print("\n" + "#" * 100) + print("# 5. IN-SAMPLE-OPTIMISED discrete rule (enter hour H, hold W, best dir).") + print("# Picked by IS Sharpe, reported OOS. Demonstrates the multiple-testing trap.") + print("#" * 100) + for a in ASSETS: + r = discrete_hour_rule_scan(data[a]) + print(f" {a}: tested {r['n_tested']} (H,W,dir) cells -> best IS " + f"H={r['H']:02d} hold={r['W']}h dir={r['dir']:+d} " + f"IS Sh={r['sh_is']:>+5.2f} (n={r['n_is']}) -> OOS Sh={r['sh_oos']:>+5.2f} " + f"(n={r['n_oos']}, mean {r['oos_mean_bp']:>+.1f} bp/trade)") + + # --- VERDICT ------------------------------------------------------------- + print("\n" + "#" * 100) + print("# MULTIPLE-TESTING CAVEAT") + print("#" * 100) + print(""" + Buckets examined: 24 hours + 7 weekdays + 168 hour×weekday = 199 calendar cells PER ASSET, + each tested IS and OOS, plus discrete grid = 24×8×2 = 384 (H,W,dir) cells per asset. + With that many cells, spurious 'significant' buckets are GUARANTEED. The honest filters + applied here: (a) adaptive sign chosen live on PAST data only (no cherry-picking), + (b) must hold OOS, (c) must hold per-year, (d) must hold on BOTH BTC AND ETH. + Read the IS->OOS Sharpe collapse and the per-year sign flips above as the real verdict. +""") + + +if __name__ == "__main__": + main() diff --git a/scripts/research/trackG_prior_levels.py b/scripts/research/trackG_prior_levels.py new file mode 100644 index 0000000..fc7c143 --- /dev/null +++ b/scripts/research/trackG_prior_levels.py @@ -0,0 +1,478 @@ +"""TRACK G — PRIOR-PERIOD LEVEL BREAKOUTS / RANGE on CLEAN BTC/ETH (Deribit mainnet). + +HONEST harness only. We test rules defined RELATIVE TO A PRIOR CALENDAR PERIOD: + * prior-DAY high/low breakout (continuation AND fade) + * opening-range breakout (first N UTC hours -> break for rest of day) + * prior-day CLOSE / gap / range-position / prior-day return-sign filter + * prior-WEEK high/low breakout + * time-anchored entries (act at a given UTC hour vs prior-day level), exit EOD/fixed/TP-SL + +The single question: on clean BTC/ETH, with a genuinely EXECUTABLE entry (direction and +price decided with data <= close[i], fill at close[i], NEVER entering at the exact level +intrabar), net of realistic Deribit fees, OOS and grid-robust on BOTH assets — +do prior-period breakouts CONTINUE (trend) or REVERT (fade)? Is there a deployable edge? + +NO LOOK-AHEAD GUARANTEES: + * Prior-period levels are built by aggregating to daily/weekly bars and SHIFTING by one + full period (shift(1) on the closed-period frame). 'Today'/'this-week' is NEVER part of + the level. The prior period is fully closed before any bar of the current period. + * Opening-range levels are used ONLY on bars AFTER the open window has fully closed. + * Direction + price decided at close[i]; fill at close[i] (harness enforces). + +Run: + uv run python scripts/research/trackG_prior_levels.py # full + uv run python scripts/research/trackG_prior_levels.py --quick # 1h only, fewer grids +""" +from __future__ import annotations + +import argparse +import sys +import time +from itertools import product +from pathlib import Path + +import numpy as np +import pandas as pd + +sys.path.insert(0, str(Path(__file__).resolve().parents[2])) + +from src.backtest.harness import load, backtest_signals, oos_split + + +# =========================================================================== +# Causal helpers +# =========================================================================== +def atr(df: pd.DataFrame, period: int = 14) -> np.ndarray: + h, l, c = df["high"].values, df["low"].values, df["close"].values + pc = np.roll(c, 1) + pc[0] = c[0] + tr = np.maximum(h - l, np.maximum(np.abs(h - pc), np.abs(l - pc))) + return pd.Series(tr).ewm(alpha=1.0 / period, adjust=False).mean().values + + +def prior_period_levels(df: pd.DataFrame, period: str = "D") -> dict: + """Return prior-period high/low/close/open/range arrays aligned to each intraday bar. + + period='D': prior calendar day (UTC). period='W': prior ISO week (anchored Mon 00:00 UTC). + Uses shift(1) on the CLOSED-period frame: the level for the current period only sees the + fully-closed previous period -> no look-ahead. + """ + dt = df["datetime"] + if period == "D": + key = dt.dt.floor("D") + elif period == "W": + key = dt.dt.floor("D") - pd.to_timedelta(dt.dt.weekday, unit="D") + else: + raise ValueError(period) + + key = key.reset_index(drop=True) + agg = pd.DataFrame({ + "key": key, + "high": df["high"].values, "low": df["low"].values, + "close": df["close"].values, "open": df["open"].values, + }) + g = agg.groupby("key").agg(high=("high", "max"), low=("low", "min"), + close=("close", "last"), open=("open", "first")).sort_index() + gp = g.shift(1) # prior, fully-closed period + km = key.map # map current-period key -> prior-period aggregate + ph = km(gp["high"]).values.astype(float) + pl = km(gp["low"]).values.astype(float) + pc = km(gp["close"]).values.astype(float) + po = km(gp["open"]).values.astype(float) + pret = (gp["close"] / gp["open"] - 1.0) # prior-period return (sign filter) + prv = key.map(pret).values.astype(float) + return {"ph": ph, "pl": pl, "pc": pc, "po": po, "prange": ph - pl, "pret": prv} + + +def opening_range(df: pd.DataFrame, n_open_hours: int) -> dict: + """Opening-range high/low for the first n_open_hours of each UTC day, plus a per-bar + flag of whether the open window has CLOSED (hour >= n_open_hours).""" + dt = df["datetime"] + date = dt.dt.floor("D") + hour = dt.dt.hour + date = date.reset_index(drop=True) + in_open = (hour < n_open_hours).values + o = pd.DataFrame({"date": date, "high": df["high"].values, "low": df["low"].values}) + o_open = o[in_open] + org = o_open.groupby("date").agg(orh=("high", "max"), orl=("low", "min")) + orh = date.map(org["orh"]).values.astype(float) + orl = date.map(org["orl"]).values.astype(float) + closed = (hour >= n_open_hours).values + return {"orh": orh, "orl": orl, "closed": closed} + + +def bars_left_in_day(df: pd.DataFrame) -> np.ndarray: + date = df["datetime"].dt.floor("D") + grp = df.groupby(date) + idx_in_day = grp.cumcount().values + size = grp["close"].transform("size").values + return (size - idx_in_day - 1).astype(int) + + +# =========================================================================== +# Signal generators -> list[dict|None] length len(df). Decisions use data <= close[i]. +# =========================================================================== +def sig_prior_break(df, period="D", level="high", side="cont", anchor_hour=None, + exit_mode="eod", max_bars=24, tp_atr=0.0, sl_atr=0.0, atr_p=14, + buffer=0.0): + """Prior-period level breakout. + level='high': trigger when close[i] > prior_high*(1+buffer) + level='low' : trigger when close[i] < prior_low *(1-buffer) + side='cont' : trade IN the breakout direction (high->long, low->short) + side='fade' : trade AGAINST it (high->short, low->long) + anchor_hour : if set, only evaluate on bars at that UTC hour (time-anchored) + exit_mode : 'eod' (close at end of UTC day), 'bars' (max_bars), TP/SL via *_atr. + """ + lv = prior_period_levels(df, period) + c = df["close"].values + a = atr(df, atr_p) if (tp_atr or sl_atr) else None + bl = bars_left_in_day(df) if exit_mode == "eod" else None + hour = df["datetime"].dt.hour.values + n = len(c) + out = [None] * n + ref = lv["ph"] if level == "high" else lv["pl"] + for i in range(n): + if anchor_hour is not None and hour[i] != anchor_hour: + continue + r = ref[i] + if not np.isfinite(r): + continue + px = c[i] + if level == "high": + if not (px > r * (1.0 + buffer)): + continue + brk_dir = 1 + else: + if not (px < r * (1.0 - buffer)): + continue + brk_dir = -1 + direction = brk_dir if side == "cont" else -brk_dir + if exit_mode == "eod": + mb = max(int(bl[i]), 1) + else: + mb = max_bars + tp = sl = None + if a is not None and np.isfinite(a[i]): + if tp_atr: + tp = px + direction * tp_atr * a[i] + if sl_atr: + sl = px - direction * sl_atr * a[i] + out[i] = {"dir": direction, "tp": tp, "sl": sl, "max_bars": mb} + return out + + +def sig_or_break(df, n_open_hours=6, side="cont", exit_mode="eod", max_bars=12, + tp_atr=0.0, sl_atr=0.0, atr_p=14, buffer=0.0): + """Opening-range breakout: after the first n_open_hours close, trade a break of the + OR high (long if cont) or OR low (short if cont). Only the FIRST break per day fires + (the harness keeps the position busy until exit).""" + orr = opening_range(df, n_open_hours) + c = df["close"].values + a = atr(df, atr_p) if (tp_atr or sl_atr) else None + bl = bars_left_in_day(df) if exit_mode == "eod" else None + n = len(c) + out = [None] * n + orh, orl, closed = orr["orh"], orr["orl"], orr["closed"] + for i in range(n): + if not closed[i] or not np.isfinite(orh[i]): + continue + px = c[i] + if px > orh[i]: + brk = 1 + elif px < orl[i]: + brk = -1 + else: + continue + direction = brk if side == "cont" else -brk + if exit_mode == "eod": + mb = max(int(bl[i]), 1) + else: + mb = max_bars + tp = sl = None + if a is not None and np.isfinite(a[i]): + if tp_atr: + tp = px + direction * tp_atr * a[i] + if sl_atr: + sl = px - direction * sl_atr * a[i] + out[i] = {"dir": direction, "tp": tp, "sl": sl, "max_bars": mb} + return out + + +def sig_gap(df, side="cont", anchor_hour=0, thr=0.0, exit_mode="eod", max_bars=24, + ret_filter=0): + """Gap vs prior-day CLOSE, evaluated at a given UTC hour (default the first bar of the + day). gap = close[i]/prior_close - 1. If gap>thr -> up-gap; gap<-thr -> down-gap. + side='cont' trades in the gap direction; 'fade' against. ret_filter: +1 only when + prior-day return positive, -1 only when negative, 0 no filter.""" + lv = prior_period_levels(df, "D") + c = df["close"].values + bl = bars_left_in_day(df) if exit_mode == "eod" else None + hour = df["datetime"].dt.hour.values + pc, pret = lv["pc"], lv["pret"] + n = len(c) + out = [None] * n + for i in range(n): + if hour[i] != anchor_hour or not np.isfinite(pc[i]): + continue + gap = c[i] / pc[i] - 1.0 + if gap > thr: + g = 1 + elif gap < -thr: + g = -1 + else: + continue + if ret_filter and np.isfinite(pret[i]): + if ret_filter > 0 and not (pret[i] > 0): + continue + if ret_filter < 0 and not (pret[i] < 0): + continue + direction = g if side == "cont" else -g + mb = max(int(bl[i]), 1) if exit_mode == "eod" else max_bars + out[i] = {"dir": direction, "tp": None, "sl": None, "max_bars": mb} + return out + + +# =========================================================================== +# Evaluation +# =========================================================================== +def run_split(df, sigfn, params, fee_rt=0.001, leverage=1.0, frac=0.65): + cut = oos_split(df, frac) + full = backtest_signals(df, sigfn(df, **params), fee_rt=fee_rt, leverage=leverage) + di = df.iloc[:cut].reset_index(drop=True) + do = df.iloc[cut:].reset_index(drop=True) + is_ = backtest_signals(di, sigfn(di, **params), fee_rt=fee_rt, leverage=leverage) + oos = backtest_signals(do, sigfn(do, **params), fee_rt=fee_rt, leverage=leverage) + return full, is_, oos + + +def hdr(t): + print("\n" + "=" * 100) + print(t) + print("=" * 100) + + +# =========================================================================== +# Main +# =========================================================================== +def main(): + ap = argparse.ArgumentParser() + ap.add_argument("--quick", action="store_true") + args = ap.parse_args() + t0 = time.time() + assets = ["BTC", "ETH"] + tfs = ["1h"] if args.quick else ["1h", "15m"] + + data = {} + hdr("DATA") + for a in assets: + for tf in tfs: + df = load(a, tf) + data[(a, tf)] = df + print(f" {a} {tf:>3s}: {len(df):>7d} bars {df['datetime'].iloc[0].date()}" + f"->{df['datetime'].iloc[-1].date()}") + + # --------------------------------------------------------------------- + # PASS 1 — PRIOR-DAY BREAKOUT: continuation vs fade, any-bar (first break/day), + # EOD exit. THE core question: do prior-day breakouts continue or revert? + # --------------------------------------------------------------------- + hdr("PASS 1 — PRIOR-DAY HIGH/LOW breakout, any-bar first-break, EOD exit (1h, fee=0.001)\n" + " CONTINUATION vs FADE side-by-side. OOS net must be >0 on BOTH to matter.") + print(f" {'rule':<26s} | " + f"{'BTC IS / OOS (tr, wr, shrp)':<40s} | {'ETH IS / OOS (tr, wr, shrp)':<40s}") + for level in ["high", "low"]: + for side in ["cont", "fade"]: + name = f"PD {level:<4s} {side}" + line = f" {name:<26s} | " + for a in assets: + df = data[(a, "1h")] + _, is_, oos = run_split(df, sig_prior_break, + dict(period="D", level=level, side=side, + exit_mode="eod")) + line += (f"{is_.net_return*100:>+6.0f}/{oos.net_return*100:>+6.0f}% " + f"(t{oos.n_trades:>4d} w{oos.win_rate:>4.1f} s{oos.sharpe:>+4.1f}) | ") + print(line) + + # --------------------------------------------------------------------- + # PASS 2 — OPENING-RANGE breakout (continuation vs fade), various open windows. + # --------------------------------------------------------------------- + hdr("PASS 2 — OPENING-RANGE breakout (first N UTC hours), EOD exit (1h, fee=0.001).\n" + " CONTINUATION vs FADE. Survivor = OOS>0 on BOTH assets.") + for nopen in ([6] if args.quick else [3, 6, 8, 12]): + for side in ["cont", "fade"]: + name = f"OR N={nopen:<2d} {side}" + line = f" {name:<26s} | " + for a in assets: + df = data[(a, "1h")] + _, is_, oos = run_split(df, sig_or_break, + dict(n_open_hours=nopen, side=side, exit_mode="eod")) + line += (f"{a} OOS={oos.net_return*100:>+6.0f}% " + f"(t{oos.n_trades:>4d} w{oos.win_rate:>4.1f} s{oos.sharpe:>+4.1f}) | ") + print(line) + + # --------------------------------------------------------------------- + # PASS 3 — GAP vs prior close at day open (hour 0), continuation vs fade, + # with optional prior-day return-sign filter. + # --------------------------------------------------------------------- + hdr("PASS 3 — GAP vs prior-day CLOSE at hour 0, EOD exit (1h, fee=0.001).\n" + " continuation vs fade; thr = min |gap|.") + for thr in ([0.0] if args.quick else [0.0, 0.005, 0.01]): + for side in ["cont", "fade"]: + name = f"GAP thr={thr*100:.1f}% {side}" + line = f" {name:<26s} | " + for a in assets: + df = data[(a, "1h")] + _, is_, oos = run_split(df, sig_gap, + dict(side=side, anchor_hour=0, thr=thr, exit_mode="eod")) + line += (f"{a} OOS={oos.net_return*100:>+6.0f}% " + f"(t{oos.n_trades:>4d} w{oos.win_rate:>4.1f} s{oos.sharpe:>+4.1f}) | ") + print(line) + + # --------------------------------------------------------------------- + # PASS 4 — PRIOR-WEEK high/low breakout (continuation vs fade), EOD exit. + # --------------------------------------------------------------------- + hdr("PASS 4 — PRIOR-WEEK HIGH/LOW breakout, any-bar first-break, EOD exit (1h, fee=0.001).") + for level in ["high", "low"]: + for side in ["cont", "fade"]: + name = f"PW {level:<4s} {side}" + line = f" {name:<26s} | " + for a in assets: + df = data[(a, "1h")] + _, is_, oos = run_split(df, sig_prior_break, + dict(period="W", level=level, side=side, + exit_mode="eod")) + line += (f"{a} IS={is_.net_return*100:>+6.0f}% OOS={oos.net_return*100:>+6.0f}% " + f"(t{oos.n_trades:>4d} s{oos.sharpe:>+4.1f}) | ") + print(line) + + # --------------------------------------------------------------------- + # PASS 5 — TIME-ANCHORED prior-day breakout: sweep the anchor hour to expose + # whether any apparent edge is just a lucky single hour. + # --------------------------------------------------------------------- + hdr("PASS 5 — TIME-ANCHORED PD-high CONTINUATION across UTC anchor hours (1h, EOD exit).\n" + " A real edge is NOT a single lucky hour. (full-sample net per hour.)") + hours = list(range(0, 24, 1 if not args.quick else 3)) + for a in assets: + df = data[(a, "1h")] + cells = [] + for hh in hours: + full, _, _ = run_split(df, sig_prior_break, + dict(period="D", level="high", side="cont", + anchor_hour=hh, exit_mode="eod")) + cells.append((hh, full.net_return * 100, full.sharpe, full.n_trades)) + pos = sum(1 for _, r, _, _ in cells if r > 0) + print(f" {a}: {pos}/{len(cells)} anchor-hours net>0 (full). " + f"best={max(cells, key=lambda x: x[1])[0]}h " + f"({max(c[1] for c in cells):+.0f}%) worst={min(c[1] for c in cells):+.0f}%") + line = " " + " ".join(f"{hh:02d}h:{r:>+5.0f}" for hh, r, _, _ in cells) + print(line) + + # --------------------------------------------------------------------- + # PASS 6 — GRID ROBUSTNESS on the best family from PASS 1-4. We grid the + # PD-low CONTINUATION and FADE plus OR breakout, require OOS>0 on BOTH assets. + # --------------------------------------------------------------------- + hdr("PASS 6 — GRID ROBUSTNESS. Cell SURVIVES only if OOS net>0 on BOTH BTC AND ETH.") + + def grid(label, fn, base, sweep, tf="1h", fee=0.001): + keys = list(sweep.keys()) + rows, surv = [], [] + for combo in product(*[sweep[k] for k in keys]): + params = dict(base); params.update(dict(zip(keys, combo))) + res = {} + for a in assets: + _, is_, oos = run_split(data[(a, tf)], fn, params, fee_rt=fee) + res[a] = oos + ok = all(res[a].net_return > 0 for a in assets) + rows.append((params, res, ok)) + if ok: + surv.append((params, res)) + print(f" [{label}] {len(surv)}/{len(rows)} cells OOS>0 on BOTH assets") + rows.sort(key=lambda r: np.mean([r[1][a].net_return for a in assets]), reverse=True) + for params, res, ok in rows[:5]: + tag = "OK " if ok else " -" + pp = {k: params[k] for k in sweep} + s = f" {tag}{pp} | " + for a in assets: + s += f"{a} OOS={res[a].net_return*100:>+6.0f}% (s{res[a].sharpe:>+4.1f}) " + print(s) + return surv + + sweeps = [] + sweeps.append(grid("PD-low cont", sig_prior_break, + dict(period="D", level="low", side="cont", exit_mode="eod"), + dict(buffer=[0.0, 0.001, 0.003], anchor_hour=[None]))) + sweeps.append(grid("PD-low fade", sig_prior_break, + dict(period="D", level="low", side="fade", exit_mode="eod"), + dict(buffer=[0.0, 0.001, 0.003], anchor_hour=[None]))) + sweeps.append(grid("PD-high cont", sig_prior_break, + dict(period="D", level="high", side="cont", exit_mode="eod"), + dict(buffer=[0.0, 0.001, 0.003], anchor_hour=[None]))) + sweeps.append(grid("PD-high fade", sig_prior_break, + dict(period="D", level="high", side="fade", exit_mode="eod"), + dict(buffer=[0.0, 0.001, 0.003], anchor_hour=[None]))) + if not args.quick: + sweeps.append(grid("OR cont", sig_or_break, + dict(side="cont", exit_mode="eod"), + dict(n_open_hours=[3, 6, 8, 12]))) + sweeps.append(grid("OR fade", sig_or_break, + dict(side="fade", exit_mode="eod"), + dict(n_open_hours=[3, 6, 8, 12]))) + + # --------------------------------------------------------------------- + # PASS 7 — FEE SWEEP + per-year on the single best surviving rule (if any), + # else on the least-bad PD rule, to show fee sensitivity and year stability. + # --------------------------------------------------------------------- + hdr("PASS 7 — FEE SWEEP + PER-YEAR on the best PD rule. fee=0 is GROSS (is the SIGN of\n" + " the edge even right before fees?).") + # pick best rule: scan the 4 PD sides at default, mean OOS over assets + candidates = [ + ("PD low cont", dict(period="D", level="low", side="cont", exit_mode="eod")), + ("PD low fade", dict(period="D", level="low", side="fade", exit_mode="eod")), + ("PD high cont", dict(period="D", level="high", side="cont", exit_mode="eod")), + ("PD high fade", dict(period="D", level="high", side="fade", exit_mode="eod")), + ] + scored = [] + for nm, p in candidates: + m = np.mean([run_split(data[(a, "1h")], sig_prior_break, p)[2].net_return for a in assets]) + scored.append((m, nm, p)) + scored.sort(reverse=True) + best_nm, best_p = scored[0][1], scored[0][2] + print(f" best-by-meanOOS PD rule: {best_nm} (meanOOS={scored[0][0]*100:+.0f}%)") + fees = [0.0, 0.0005, 0.001, 0.0015, 0.002] + for a in assets: + df = data[(a, "1h")] + line = f" {a} fee-sweep (RT%): " + for f in fees: + full, _, oos = run_split(df, sig_prior_break, best_p, fee_rt=f) + line += f"{f*100:.2f}%[full={full.net_return*100:>+5.0f}/OOS={oos.net_return*100:>+5.0f}] " + print(line) + print(" per-year (full sample, fee=0.001):") + for a in assets: + df = data[(a, "1h")] + full, _, _ = run_split(df, sig_prior_break, best_p) + yrs = " ".join(f"{y}:{full.yearly[y]*100:>+5.0f}%" for y in sorted(full.yearly)) + print(f" {a}: trades={full.n_trades} Sharpe={full.sharpe:+.2f} " + f"maxDD={full.max_dd*100:.0f}% EUR/d(2k)={full.daily_profit(2000):+.2f}") + print(f" {yrs}") + + # --------------------------------------------------------------------- + # VERDICT + # --------------------------------------------------------------------- + hdr("VERDICT") + total_surv = sum(len(s) for s in sweeps) + if total_surv == 0: + print(" ZERO grid cells produced OOS net>0 on BOTH BTC and ETH at baseline fees.") + print(" => No robust prior-period breakout/fade edge on clean BTC/ETH. The continuation-") + print(" vs-fade tables above show which SIDE (if any) is even net-positive in-sample;") + print(" consult PASS 1-5 for direction. Not deployable.") + else: + print(f" {total_surv} grid cell(s) survived OOS>0 on both assets. Inspect PASS 6/7 and") + print(" stress with fee sweep + per-year before trusting. List of survivors:") + for s in sweeps: + for params, res in s: + ms = np.mean([res[a].net_return for a in assets]) * 100 + print(f" {params} meanOOS={ms:+.0f}%") + print(f"\n (elapsed {time.time()-t0:.0f}s)") + + +if __name__ == "__main__": + main() diff --git a/scripts/research/trackH_volume_vol.py b/scripts/research/trackH_volume_vol.py new file mode 100644 index 0000000..04fcb1f --- /dev/null +++ b/scripts/research/trackH_volume_vol.py @@ -0,0 +1,602 @@ +"""TRACK H — VOLUME, RANGE & VOLATILITY-REGIME signals on CLEAN BTC/ETH (Deribit mainnet). + +The single question: net of realistic Deribit fees, OOS-robust on BOTH BTC & ETH, on >=12h +timeframes (the only honest regime — sub-12h is fees + HF-noise overfit + the open-label +look-ahead trap), is there ANY volume / range / volatility-regime signal that is + + (a) net-positive OOS on both assets standalone, AND + (b) uncorrelated (|corr| < ~0.3) to the deployed winner TP01, AND/OR + (c) usable as a REGIME FILTER that lifts TP01's Sharpe above ~1.32 or cuts its DD? + +HONESTY / NO LOOK-AHEAD: + * Everything runs on the SAME causal per-bar engine used by TP01 (net_returns): we build a + continuous TARGET position decided with data <= close[i], then HOLD it during bar i+1 + (pos_held[t] = target[t-1]). Gross = pos_held * simple_return[t]; fee charged on |Δpos|. + This is identical in spirit to the harness `backtest_signals` (decide<=close[i], fill at + close[i]); we cross-check two discrete signals through `backtest_signals` too. + * Volume / range / vol features for bar i use ONLY bars <= i (rolling, prior-window, shift). + * 12h / 1d frames are resampled from the certified 1h feed via resample_tf (label='left', + closed='left') and consumed index-based with the +1 bar hold -> the open-label is never + leaked (verified in trackD_lookahead_audit.py: Sharpe is label-invariant under this hold). + +Run: + uv run python scripts/research/trackH_volume_vol.py # full (12h + 1d) + uv run python scripts/research/trackH_volume_vol.py --quick # 12h only, fewer grids +""" +from __future__ import annotations + +import argparse +import sys +from pathlib import Path + +import numpy as np +import pandas as pd + +sys.path.insert(0, str(Path(__file__).resolve().parents[2])) + +from src.backtest.harness import load, backtest_signals +from src.strategies.trend_portfolio import TrendPortfolio, CANONICAL, resample_tf + +ASSETS = ["BTC", "ETH"] +FEE_SIDE = 0.0005 # 0.05%/side = 0.10% RT (Deribit taker) +OOS_FRAC = 0.65 +TF_BPD = {"12h": 2, "1d": 1} + + +# =========================================================================== +# Causal feature helpers (all use data <= i) +# =========================================================================== +def simple_returns(c: np.ndarray) -> np.ndarray: + r = np.zeros(len(c)) + r[1:] = c[1:] / c[:-1] - 1.0 + return r + + +def realized_vol(r: np.ndarray, win: int, bpy: float) -> np.ndarray: + return pd.Series(r).rolling(win, min_periods=max(2, win // 2)).std().values * np.sqrt(bpy) + + +def roll_max_prior(x: np.ndarray, win: int) -> np.ndarray: + """Max over the PRIOR `win` bars (excludes current bar i).""" + return pd.Series(x).shift(1).rolling(win, min_periods=win).max().values + + +def roll_min_prior(x: np.ndarray, win: int) -> np.ndarray: + return pd.Series(x).shift(1).rolling(win, min_periods=win).min().values + + +def roll_mean_prior(x: np.ndarray, win: int) -> np.ndarray: + return pd.Series(x).shift(1).rolling(win, min_periods=win).mean().values + + +def vol_zscore(vol: np.ndarray, win: int) -> np.ndarray: + """z-score of current volume vs PRIOR `win` bars (uses <= i).""" + s = pd.Series(vol) + m = s.shift(1).rolling(win, min_periods=win).mean() + sd = s.shift(1).rolling(win, min_periods=win).std() + return ((s - m) / sd).values + + +def atr(df: pd.DataFrame, period: int) -> np.ndarray: + h, l, c = df["high"].values, df["low"].values, df["close"].values + pc = np.roll(c, 1) + pc[0] = c[0] + tr = np.maximum(h - l, np.maximum(np.abs(h - pc), np.abs(l - pc))) + return pd.Series(tr).ewm(alpha=1.0 / period, adjust=False).mean().values + + +# =========================================================================== +# Per-bar net-returns engine (causal, fee on turnover) — identical to TP01.net_returns +# =========================================================================== +def net_from_target(target: np.ndarray, r: np.ndarray, fee_side: float): + """target[i] decided with data <= close[i] -> HELD during bar i+1.""" + target = np.nan_to_num(target, nan=0.0) + pos = np.zeros(len(target)) + pos[1:] = target[:-1] + gross = pos * r + turn = np.abs(np.diff(pos, prepend=0.0)) + net = gross - fee_side * turn + net[0] = 0.0 + net = np.clip(net, -0.99, None) + return net, pos, turn + + +def metrics(net: np.ndarray, idx: pd.DatetimeIndex, turn: np.ndarray, bpy: float) -> dict: + rr = net[np.isfinite(net)] + sharpe = float(np.mean(rr) / np.std(rr) * np.sqrt(bpy)) if np.std(rr) > 0 else 0.0 + equity = np.cumprod(1.0 + np.clip(net, -0.99, None)) + peak = np.maximum.accumulate(equity) + dd = float(np.max((peak - equity) / peak)) if len(equity) else 0.0 + span_days = (idx[-1] - idx[0]).total_seconds() / 86400 + years = span_days / 365.25 if span_days > 0 else 1.0 + total = equity[-1] / equity[0] if len(equity) else 1.0 + cagr = total ** (1 / years) - 1 if years > 0 and total > 0 else -1.0 + ann_turn = float(np.sum(turn)) / years if years > 0 else 0.0 + return dict(sharpe=sharpe, max_dd=dd, cagr=cagr, total=total - 1, + ann_turnover=ann_turn, equity=equity, years=years) + + +def per_year(net: np.ndarray, idx: pd.DatetimeIndex) -> dict: + eq = pd.Series(np.cumprod(1.0 + np.clip(net, -0.99, None)), index=idx) + out = {} + for y, g in eq.groupby(eq.index.year): + if len(g) > 1 and g.iloc[0] > 0: + out[int(y)] = float(g.iloc[-1] / g.iloc[0] - 1) + return out + + +# =========================================================================== +# SIGNALS — each returns a continuous TARGET array (frac of equity, +/-), causal. +# =========================================================================== +def sig_vt_long(df, bpd, target_vol=0.20, vol_win_days=30, lev=2.0, **_): + """Volatility-managed LONG: always long, sized to a vol target (no trend at all). + Tests Moreira-Muir 'volatility-managed' alpha vs plain buy-and-hold.""" + c = df["close"].values.astype(float) + r = simple_returns(c) + bpy = bpd * 365.25 + vol = realized_vol(r, vol_win_days * bpd, bpy) + tgt = np.where((vol > 0) & np.isfinite(vol), target_vol / vol, 0.0) + return np.clip(tgt, 0, lev) + + +def sig_vol_breakout(df, bpd, don=20, zwin=20, zk=1.0, long_short=False, **_): + """Volume-confirmed Donchian breakout (continuation). Long when close > prior-`don`-bar high + AND volume z-score > zk; stay long until close < prior-`don`-bar low (then flat/short).""" + c = df["close"].values.astype(float) + h = df["high"].values.astype(float) + l = df["low"].values.astype(float) + vol = df["volume"].values.astype(float) + hi = roll_max_prior(h, don) + lo = roll_min_prior(l, don) + z = vol_zscore(vol, zwin) + up = (c > hi) & (z > zk) + dn = (c < lo) & (z > zk) + state = np.zeros(len(c)) + s = 0.0 + for i in range(len(c)): + if up[i]: + s = 1.0 + elif dn[i]: + s = -1.0 if long_short else 0.0 + elif s == 1.0 and c[i] < lo[i]: # trailing exit for longs + s = -1.0 if long_short else 0.0 + elif s == -1.0 and c[i] > hi[i]: + s = 1.0 + state[i] = s + return state + + +def sig_obv_trend(df, bpd, ma=30, long_short=False, **_): + """OBV trend: OBV = cumsum(sign(ret)*volume); long when OBV > its EMA(ma), else flat/short.""" + c = df["close"].values.astype(float) + vol = df["volume"].values.astype(float) + r = simple_returns(c) + obv = np.cumsum(np.sign(r) * vol) + ema = pd.Series(obv).ewm(span=ma, adjust=False).mean().values + d = np.where(obv > ema, 1.0, (-1.0 if long_short else 0.0)) + return d + + +def sig_vw_momentum(df, bpd, mom_win=30, vol_win_days=30, target_vol=0.20, lev=2.0, + long_only=True, **_): + """Volume-weighted momentum: sign of volume-weighted mean return over `mom_win` bars, + vol-targeted. Compare to plain TSMOM (does weighting by volume add anything?).""" + c = df["close"].values.astype(float) + vol = df["volume"].values.astype(float) + r = simple_returns(c) + rw = r * vol + num = pd.Series(rw).rolling(mom_win, min_periods=mom_win).sum().values + den = pd.Series(vol).rolling(mom_win, min_periods=mom_win).sum().values + vwret = np.where(den > 0, num / den, 0.0) + direction = np.sign(vwret) + if long_only: + direction = np.clip(direction, 0, None) + bpy = bpd * 365.25 + rv = realized_vol(r, vol_win_days * bpd, bpy) + scal = np.where((rv > 0) & np.isfinite(rv), target_vol / rv, 0.0) + return np.clip(direction * scal, -lev, lev) + + +def sig_range_expansion(df, bpd, rng_win=20, k=1.5, hold=5, long_short=False, **_): + """Range-expansion breakout: when today's range > k * avg(prior `rng_win` ranges) and the + bar closed in the upper/lower half, go with the close direction; hold `hold` bars.""" + c = df["close"].values.astype(float) + h = df["high"].values.astype(float) + l = df["low"].values.astype(float) + rng = h - l + avg = roll_mean_prior(rng, rng_win) + expand = rng > k * avg + pos_in_bar = np.where(rng > 0, (c - l) / rng, 0.5) + long_trig = expand & (pos_in_bar > 0.6) + short_trig = expand & (pos_in_bar < 0.4) + state = np.zeros(len(c)) + hold_left = 0 + cur = 0.0 + for i in range(len(c)): + if hold_left > 0: + hold_left -= 1 + else: + cur = 0.0 + if long_trig[i]: + cur = 1.0 + hold_left = hold + elif short_trig[i] and long_short: + cur = -1.0 + hold_left = hold + state[i] = cur + return state + + +def sig_nr_breakout(df, bpd, nr=7, hold=5, long_short=False, **_): + """NR-N breakout (daily-style): when the current bar's range is the narrowest of the last + `nr` bars, take the breakout of the prior bar's high/low on the next bars; hold `hold`.""" + c = df["close"].values.astype(float) + h = df["high"].values.astype(float) + l = df["low"].values.astype(float) + rng = h - l + is_nr = pd.Series(rng).rolling(nr, min_periods=nr).apply( + lambda w: 1.0 if w[-1] == np.min(w) else 0.0, raw=True).values + state = np.zeros(len(c)) + cur = 0.0 + hold_left = 0 + armed = False + arm_hi = arm_lo = np.nan + for i in range(len(c)): + if hold_left > 0: + hold_left -= 1 + else: + cur = 0.0 + if armed: + if c[i] > arm_hi: + cur = 1.0 + hold_left = hold + armed = False + elif c[i] < arm_lo and long_short: + cur = -1.0 + hold_left = hold + armed = False + if is_nr[i] == 1.0: + armed = True + arm_hi = h[i] + arm_lo = l[i] + state[i] = cur + return state + + +def sig_decl_vol_reversal(df, bpd, mom_win=10, vwin=10, **_): + """Declining-volume reversal (fade): after an up-move on DECLINING volume, fade (short); + after a down-move on declining volume, go long. Pure contrarian, vol-confirmed exhaustion.""" + c = df["close"].values.astype(float) + vol = df["volume"].values.astype(float) + ret = pd.Series(c).pct_change(mom_win).values + vtrend = vol - roll_mean_prior(vol, vwin) + declining = vtrend < 0 + state = np.zeros(len(c)) + state[(ret > 0) & declining] = -1.0 + state[(ret < 0) & declining] = 1.0 + return state + + +SIGNALS = { + "VT-long": (sig_vt_long, dict(target_vol=0.20, vol_win_days=30, lev=2.0)), + "VolBreakout": (sig_vol_breakout, dict(don=20, zwin=20, zk=1.0)), + "OBV-trend": (sig_obv_trend, dict(ma=30)), + "VW-mom": (sig_vw_momentum, dict(mom_win=30, vol_win_days=30)), + "RangeExpand": (sig_range_expansion, dict(rng_win=20, k=1.5, hold=5)), + "NR7-break": (sig_nr_breakout, dict(nr=7, hold=5)), + "DeclVolRev": (sig_decl_vol_reversal, dict(mom_win=10, vwin=10)), +} + + +# =========================================================================== +# Evaluation +# =========================================================================== +def eval_signal(fn, params, tf, asset, fee_side=FEE_SIDE): + df = resample_tf(load(asset, "1h"), tf) + bpd = TF_BPD[tf] + bpy = bpd * 365.25 + c = df["close"].values.astype(float) + r = simple_returns(c) + idx = pd.to_datetime(df["datetime"].values) + tgt = fn(df, bpd, **params) + net, pos, turn = net_from_target(tgt, r, fee_side) + m = metrics(net, idx, turn, bpy) + # OOS split + cut = int(len(net) * OOS_FRAC) + mi = metrics(net[:cut], idx[:cut], turn[:cut], bpy) + mo = metrics(net[cut:], idx[cut:], turn[cut:], bpy) + return dict(net=net, idx=idx, full=m, is_=mi, oos=mo, py=per_year(net, idx)) + + +def tp01_net(asset, tf): + tp = TrendPortfolio(**CANONICAL) + df = resample_tf(load(asset, "1h"), tf) + net, ts = tp.net_returns(df) + return pd.Series(net, index=pd.to_datetime(ts.values)) + + +def corr_to_tp01(net, idx, tp_series): + s = pd.Series(net, index=idx) + j = pd.concat([s.rename("a"), tp_series.rename("b")], axis=1, join="inner").fillna(0.0) + if j["a"].std() == 0 or j["b"].std() == 0: + return 0.0 + return float(j["a"].corr(j["b"])) + + +# =========================================================================== +# Reports +# =========================================================================== +def report_headline(tf, quick): + print("\n" + "=" * 120) + print(f"# HEADLINE — TF {tf} | standalone signals, full / IS / OOS, turnover, corr→TP01 (fee 0.10% RT)") + print("=" * 120) + tp = {a: tp01_net(a, tf) for a in ASSETS} + print(f" {'signal':<14s}{'asset':<6s}" + f"{'fullShrp':>9s}{'fullCAGR':>9s}{'fullDD':>7s}" + f"{'IS_Shrp':>8s}{'OOS_Shrp':>9s}{'OOS_ret':>8s}{'turn/y':>8s}{'corrTP':>8s}") + results = {} + for name, (fn, params) in SIGNALS.items(): + for a in ASSETS: + res = eval_signal(fn, params, tf, a) + cr = corr_to_tp01(res["net"], res["idx"], tp[a]) + results[(name, a)] = (res, cr) + print(f" {name:<14s}{a:<6s}" + f"{res['full']['sharpe']:>9.2f}{res['full']['cagr']*100:>8.1f}%" + f"{res['full']['max_dd']*100:>6.1f}%" + f"{res['is_']['sharpe']:>8.2f}{res['oos']['sharpe']:>9.2f}" + f"{res['oos']['total']*100:>7.1f}%{res['full']['ann_turnover']:>8.1f}{cr:>8.2f}") + return results, tp + + +def report_peryear(results): + print("\n" + "-" * 120) + print("# PER-YEAR net return (%) — only signals with OOS Sharpe>0 on BOTH assets shown") + print("-" * 120) + years = list(range(2018, 2027)) + # which signals pass OOS>0 both assets + good = [] + for name in SIGNALS: + if all(results[(name, a)][0]["oos"]["sharpe"] > 0 for a in ASSETS): + good.append(name) + if not good: + print(" (none — no signal has positive OOS Sharpe on BOTH assets)") + return good + print(" " + " " * 22 + "".join(f"{y:>7d}" for y in years)) + for name in good: + for a in ASSETS: + py = results[(name, a)][0]["py"] + row = "".join((" . " if y not in py else f"{py[y]*100:>+7.0f}") for y in years) + print(f" {name+' '+a:<22s}{row}") + return good + + +def report_grid(quick): + print("\n" + "=" * 120) + print("# GRID ROBUSTNESS (TF 12h) — fraction of cells with positive full+OOS Sharpe on BOTH assets") + print("=" * 120) + tf = "12h" + grids = { + "VolBreakout": ("sig", sig_vol_breakout, + dict(don=[10, 20, 40] if not quick else [20], + zwin=[10, 20, 40], zk=[0.5, 1.0, 2.0])), + "OBV-trend": ("sig", sig_obv_trend, dict(ma=[15, 30, 60, 100])), + "VW-mom": ("sig", sig_vw_momentum, + dict(mom_win=[15, 30, 60, 90], long_only=[True])), + "RangeExpand": ("sig", sig_range_expansion, + dict(rng_win=[10, 20, 40], k=[1.3, 1.5, 2.0], hold=[3, 5, 10])), + "VT-long": ("sig", sig_vt_long, dict(target_vol=[0.15, 0.20, 0.30], + vol_win_days=[15, 30, 60])), + } + from itertools import product + for name, (_, fn, axes) in grids.items(): + keys = list(axes.keys()) + combos = list(product(*[axes[k] for k in keys])) + npos = 0 + best = (-9, None) + for combo in combos: + params = dict(zip(keys, combo)) + ok = True + sh_sum = 0.0 + for a in ASSETS: + res = eval_signal(fn, params, tf, a) + if not (res["full"]["sharpe"] > 0 and res["oos"]["sharpe"] > 0): + ok = False + sh_sum += res["oos"]["sharpe"] + if ok: + npos += 1 + if sh_sum > best[0]: + best = (sh_sum, params) + print(f" {name:<14s} positive both(full&OOS): {npos:>3d}/{len(combos):<3d} " + f"({npos/len(combos)*100:>4.0f}%) best OOS-sum cfg: {best[1]}") + + +def report_feesweep(): + print("\n" + "=" * 120) + print("# FEE SWEEP (TF 12h) — OOS Sharpe (BTC/ETH) vs round-trip fee for the headline signals") + print("=" * 120) + tf = "12h" + fees = [0.0, 0.0005, 0.001, 0.0015, 0.002] # per side; RT = 2x + print(f" {'signal':<14s}" + "".join(f" RT{f*2*100:>4.2f}%" for f in fees)) + for name, (fn, params) in SIGNALS.items(): + cells = [] + for f in fees: + shs = [] + for a in ASSETS: + res = eval_signal(fn, params, tf, a, fee_side=f) + shs.append(res["oos"]["sharpe"]) + cells.append(f"{shs[0]:>4.2f}/{shs[1]:<4.2f}") + print(f" {name:<14s}" + "".join(f" {c:>9s}" for c in cells)) + + +# =========================================================================== +# REGIME FILTER on TP01 — does a vol/volume regime mask lift Sharpe or cut DD? +# =========================================================================== +def vol_regime_mask(df, bpd, win_days=30, mode="low", q=0.5): + """Boolean per-bar mask (decided <= close[i]) for a realized-vol regime. + mode='low': keep exposure when vol <= rolling median; 'high': when vol > median.""" + c = df["close"].values.astype(float) + r = simple_returns(c) + bpy = bpd * 365.25 + vol = realized_vol(r, win_days * bpd, bpy) + # causal expanding/rolling quantile threshold (use a long rolling window, prior bars) + thr = pd.Series(vol).shift(1).rolling(180 * bpd, min_periods=30 * bpd).quantile(q).values + if mode == "low": + mask = vol <= thr + else: + mask = vol > thr + return np.nan_to_num(mask.astype(float), nan=1.0) # default keep before warmup + + +def vol_managed_mask(df, bpd, win_days=30, target_vol=0.20, cap=1.5): + """Continuous vol-scaling multiplier on TP01: scale exposure by target_vol/realized_vol, + capped — an explicit volatility-managed overlay distinct from TP01's own sizing.""" + c = df["close"].values.astype(float) + r = simple_returns(c) + bpy = bpd * 365.25 + vol = realized_vol(r, win_days * bpd, bpy) + mult = np.where((vol > 0) & np.isfinite(vol), target_vol / vol, 1.0) + return np.clip(mult, 0.0, cap) + + +def report_regime_filter(tf="12h"): + print("\n" + "=" * 120) + print(f"# REGIME FILTER on TP01 (TF {tf}) — apply a vol mask/overlay to TP01 target, 50/50 portfolio") + print("=" * 120) + bpd = TF_BPD[tf] + bpy = bpd * 365.25 + tp = TrendPortfolio(**CANONICAL) + + def portfolio(transform): + """transform(df,target)->target'; returns combined 50/50 net series + idx.""" + series = {} + for a in ASSETS: + df = resample_tf(load(a, "1h"), tf) + r = simple_returns(df["close"].values.astype(float)) + tgt = tp.target_series(df) + tgt2 = transform(df, tgt) + net, _, _ = net_from_target(tgt2, r, CANONICAL["fee_side"]) + series[a] = pd.Series(net, index=pd.to_datetime(df["datetime"].values)) + J = pd.concat(series, axis=1, join="inner").fillna(0.0) + combo = 0.5 * J[ASSETS[0]].values + 0.5 * J[ASSETS[1]].values + return combo, J.index + + variants = { + "TP01 baseline": lambda df, t: t, + "× keep LOW-vol": lambda df, t: t * vol_regime_mask(df, bpd, mode="low", q=0.5), + "× keep HIGH-vol": lambda df, t: t * vol_regime_mask(df, bpd, mode="high", q=0.5), + "× keep LOW-vol q.7": lambda df, t: t * vol_regime_mask(df, bpd, mode="low", q=0.7), + "× vol-managed x1.5": lambda df, t: t * vol_managed_mask(df, bpd, cap=1.5) / + np.maximum(vol_managed_mask(df, bpd, cap=1.5).mean(), 1e-9), + "× obv-up only": lambda df, t: t * (np.where( + np.cumsum(np.sign(simple_returns(df['close'].values.astype(float))) * df['volume'].values) + > pd.Series(np.cumsum(np.sign(simple_returns(df['close'].values.astype(float))) + * df['volume'].values)).ewm(span=30, adjust=False).mean().values, + 1.0, 0.0)), + } + print(f" {'variant':<22s}{'fullShrp':>9s}{'IS_Shrp':>8s}{'OOS_Shrp':>9s}" + f"{'CAGR':>8s}{'maxDD':>8s}{'turn/y':>9s}") + for name, tr in variants.items(): + combo, idx = portfolio(tr) + m = metrics(combo, idx, np.zeros_like(combo), bpy) + cut = int(len(combo) * OOS_FRAC) + mi = metrics(combo[:cut], idx[:cut], np.zeros_like(combo[:cut]), bpy) + mo = metrics(combo[cut:], idx[cut:], np.zeros_like(combo[cut:]), bpy) + tt = 0.0 + for a in ASSETS: + df = resample_tf(load(a, "1h"), tf) + tgt2 = tr(df, tp.target_series(df)) + tt += np.sum(np.abs(np.diff(np.nan_to_num(tgt2), prepend=0.0))) + ann_tt = tt / m["years"] / 2.0 + print(f" {name:<22s}{m['sharpe']:>9.2f}{mi['sharpe']:>8.2f}{mo['sharpe']:>9.2f}" + f"{m['cagr']*100:>7.1f}%{m['max_dd']*100:>7.1f}%{ann_tt:>9.1f}") + + # robustness of the OBV-up filter across EMA spans (is 1.49 luck or stable?) + print("\n OBV-up filter robustness across EMA span (full / OOS Sharpe, maxDD):") + for span in [15, 20, 30, 45, 60, 90]: + def tr(df, t, sp=span): + c = df['close'].values.astype(float) + v = df['volume'].values.astype(float) + obv = np.cumsum(np.sign(simple_returns(c)) * v) + ema = pd.Series(obv).ewm(span=sp, adjust=False).mean().values + return t * np.where(obv > ema, 1.0, 0.0) + combo, idx = portfolio(tr) + m = metrics(combo, idx, np.zeros_like(combo), bpy) + cut = int(len(combo) * OOS_FRAC) + mo = metrics(combo[cut:], idx[cut:], np.zeros_like(combo[cut:]), bpy) + py = per_year(combo, idx) + neg_years = sum(1 for y, v in py.items() if v < 0) + print(f" span {span:>3d}: full {m['sharpe']:>4.2f} OOS {mo['sharpe']:>4.2f} " + f"DD {m['max_dd']*100:>4.1f}% CAGR {m['cagr']*100:>5.1f}% neg-years {neg_years}") + + +def main(): + ap = argparse.ArgumentParser() + ap.add_argument("--quick", action="store_true") + args = ap.parse_args() + + print("#" * 120) + print("# TRACK H — VOLUME / RANGE / VOLATILITY-REGIME on certified BTC/ETH (Deribit mainnet)") + print("# Honest engine: target decided <=close[i], held bar i+1; fee on |Δpos|; OOS 65/35; >=12h only.") + print("#" * 120) + + tfs = ["12h"] if args.quick else ["12h", "1d"] + for tf in tfs: + results, tp = report_headline(tf, args.quick) + report_peryear(results) + if tf == "12h": + crosscheck_backtest_signals() + report_grid(args.quick) + report_feesweep() + report_regime_filter("12h") + + print("\n" + "#" * 120) + print("# VERDICT (track H) — honest reading of the tables above") + print("#" * 120) + for line in [ + "1. NO uncorrelated additive edge found. Every PROFITABLE volume/range/vol signal", + " (VolBreakout, OBV-trend, VW-mom, VT-long) is trend-in-disguise: corr-to-TP01 0.61-0.75.", + " They do not diversify TP01 -> cannot raise the 50/50 portfolio Sharpe.", + "2. The genuinely LOWER-corr signals (RangeExpand ~0.48, NR7 ~0.48) FAIL OOS on >=1 asset", + " (NR7 ETH OOS Sharpe ~0.0/-0.03; RangeExpand BTC weak, ETH negative on 1d). Not deployable.", + "3. Declining-volume / fade (mean-reversion) is firmly NEGATIVE net of fees on both assets", + " and at ZERO fee -> confirms the v2.0.0 lesson: MR edge was feed contamination, it is dead.", + "4. Vol-REGIME gating of TP01 (keep low-vol / keep high-vol) HURTS Sharpe (1.32 -> 0.94/0.98).", + " A vol-managed x1.5 overlay leaves Sharpe ~flat (1.33) but raises DD (17.9%). No win.", + "5. The ONLY non-harmful overlay is an OBV-up trend-CONFIRMATION filter (keep TP01 long only", + " while OBV>EMA): full Sharpe 1.32->1.49, maxDD 13.3%->10.1%, but CAGR 16.2%->14.4%, turnover", + " +60%, and OOS gain is marginal (0.90->1.04) and span-sensitive (fades for EMA>45). It is", + " trend double-confirmation (de-risking), NOT new alpha. Worth noting as a DEFENSIVE overlay", + " if cutting DD matters more than CAGR; it does NOT robustly raise the portfolio Sharpe.", + "BOTTOM LINE: the ~1.3 portfolio-Sharpe ceiling on BTC/ETH-only HOLDS. Volume/range/vol add", + "nothing uncorrelated. TP01 stays the deployable winner.", + ]: + print(" " + line) + print("#" * 120) + + +def crosscheck_backtest_signals(): + """Cross-check two DISCRETE signals through the canonical harness `backtest_signals` + (decide<=close[i], fill at close[i]) to confirm the per-bar engine isn't flattering them.""" + print("\n" + "-" * 120) + print("# CROSS-CHECK via harness.backtest_signals (discrete entries, fee 0.10% RT, TF 12h)") + print("-" * 120) + tf = "12h" + for a in ASSETS: + df = resample_tf(load(a, "1h"), tf) + h = df["high"].values.astype(float) + l = df["low"].values.astype(float) + c = df["close"].values.astype(float) + rng = h - l + avg = roll_mean_prior(rng, 20) + pos_in_bar = np.where(rng > 0, (c - l) / rng, 0.5) + expand = rng > 1.5 * avg + entries = [None] * len(df) + for i in range(len(df)): + if expand[i] and pos_in_bar[i] > 0.6: + entries[i] = dict(dir=1, tp=None, sl=None, max_bars=5) + m = backtest_signals(df, entries, fee_rt=0.001, leverage=1.0, asset=a, tf=tf) + m.print_summary(f"RangeExpand(L,5b) {a}") + + +if __name__ == "__main__": + main() diff --git a/scripts/research/trackI_momentum_reversal.py b/scripts/research/trackI_momentum_reversal.py new file mode 100644 index 0000000..de2ab40 --- /dev/null +++ b/scripts/research/trackI_momentum_reversal.py @@ -0,0 +1,420 @@ +"""TRACK I — ALTERNATIVE MOMENTUM FORMULATIONS + LONG-HORIZON REVERSAL (BTC & ETH, >=12h). + +Goal: + (A) Find a momentum formulation that BEATS or DIVERSIFIES the canonical TP01 sign-blend + (TSMOM 1-3-6m, vol-targeted, 50/50 BTC+ETH, 12h, Sharpe ~1.32). + (B) Test the classic LONG-HORIZON REVERSAL effect (fade 12/18/24-month winners) as a + potentially UNCORRELATED positive overlay, and a momentum+reversal blend. + +Honest harness (mirrors src/strategies/trend_portfolio.py exactly): + - direction decided with data <= close[i]; positions HELD next bar (pos_held[1:] = tgt[:-1]); + - vol-target by inverse PAST-ONLY realized vol (target_vol/vol), leverage-capped; + - NET fees 0.10% RT (0.05%/side) on turnover; fee sweep included; + - 12h / 1d only (sub-12h is dominated by costs/overfit and a prior 4h look-ahead bug); + - OOS 65/35 split + per-year; robustness across lookbacks AND both assets; + - correlation vs TP01 net returns reported for EVERY candidate. + +A candidate is INTERESTING only if net-positive OOS on BOTH assets AND either + (higher portfolio Sharpe than TP01 ~1.32) OR (|corr to TP01| < ~0.3 and positive). + +Run: uv run python scripts/research/trackI_momentum_reversal.py +""" +from __future__ import annotations + +import sys +from pathlib import Path + +import numpy as np +import pandas as pd + +sys.path.insert(0, str(Path(__file__).resolve().parents[2])) + +from src.backtest.harness import load +from src.strategies.trend_portfolio import resample_tf, simple_returns, realized_vol + +ASSETS = ["BTC", "ETH"] +FEE_SIDE = 0.0005 # 0.05%/side = 0.10% RT +TARGET_VOL = 0.20 +LEVERAGE = 2.0 +VOL_WIN_DAYS = 30 +OOS_FRAC = 0.65 +MONTH = 30 # days per "month" (calendar-consistent across TFs) + +# tf -> bars_per_day +TF_BPD = {"12h": 2, "1d": 1} + + +# --------------------------------------------------------------------------- +# data +# --------------------------------------------------------------------------- +def get_df(asset: str, tf: str) -> pd.DataFrame: + df = load(asset, "1h") + rule = {"12h": "12h", "1d": "1D"}[tf] + return resample_tf(df, rule) + + +# --------------------------------------------------------------------------- +# vol-target machinery (identical convention to TP01) +# --------------------------------------------------------------------------- +def build_target(direction, vol, long_only): + d = np.clip(direction, 0, None) if long_only else direction + scal = np.where((vol > 0) & np.isfinite(vol), TARGET_VOL / vol, 0.0) + tgt = np.clip(d * scal, -LEVERAGE, LEVERAGE) + tgt[~np.isfinite(tgt)] = 0.0 + return tgt + + +def net_from_target(tgt, r, fee_side=FEE_SIDE): + pos_held = np.zeros(len(tgt)) + pos_held[1:] = tgt[:-1] + gross = pos_held * r + turn = np.abs(np.diff(pos_held, prepend=0.0)) + net = gross - fee_side * turn + net[0] = 0.0 + return np.clip(net, -0.99, None) + + +# --------------------------------------------------------------------------- +# DIRECTION FORMULATIONS (each returns array in roughly [-1, 1], causal, decided <= close[i]) +# --------------------------------------------------------------------------- +def _log_mom(c, h): + """log return over h bars; nan before h.""" + m = np.full(len(c), np.nan) + m[h:] = np.log(c[h:] / c[:-h]) + return m + + +def dir_signblend(c, bpd, horizons_m=(1, 3, 6)): + """TP01 baseline: mean of sign(log return) over horizons.""" + n = len(c) + acc = np.zeros(n); cnt = np.zeros(n) + for hm in horizons_m: + h = hm * MONTH * bpd + s = np.full(n, np.nan) + s[h:] = np.sign(c[h:] / c[:-h] - 1.0) + v = np.isfinite(s); acc[v] += s[v]; cnt[v] += 1 + out = np.zeros(n); nz = cnt > 0; out[nz] = acc[nz] / cnt[nz] + return out + + +def dir_zscore(c, bpd, horizons_m=(1, 3, 6), std_win_m=12): + """(i) Continuous momentum: z-scored cumulative log-return, tanh-bounded, multi-horizon avg.""" + n = len(c); w = std_win_m * MONTH * bpd + acc = np.zeros(n); cnt = np.zeros(n) + for hm in horizons_m: + h = hm * MONTH * bpd + m = _log_mom(c, h) + s = pd.Series(m) + sd = s.rolling(w, min_periods=w // 3).std().values + z = np.where((sd > 0) & np.isfinite(sd), m / sd, np.nan) + d = np.tanh(z) + v = np.isfinite(d); acc[v] += d[v]; cnt[v] += 1 + out = np.zeros(n); nz = cnt > 0; out[nz] = acc[nz] / cnt[nz] + return out + + +def dir_riskadj(c, bpd, horizons_m=(1, 3, 6)): + """(ii) Risk-adjusted momentum: h-horizon return / vol-of-that-horizon, tanh, multi-horizon.""" + n = len(c); r = simple_returns(c) + acc = np.zeros(n); cnt = np.zeros(n) + for hm in horizons_m: + h = hm * MONTH * bpd + ret = np.full(n, np.nan); ret[h:] = c[h:] / c[:-h] - 1.0 + # vol of the h-bar return = per-bar std over last h bars * sqrt(h) + sd = pd.Series(r).rolling(h, min_periods=h // 2).std().values * np.sqrt(h) + ra = np.where((sd > 0) & np.isfinite(sd), ret / sd, np.nan) + d = np.tanh(ra) + v = np.isfinite(d); acc[v] += d[v]; cnt[v] += 1 + out = np.zeros(n); nz = cnt > 0; out[nz] = acc[nz] / cnt[nz] + return out + + +def _ema(c, span): + return pd.Series(c).ewm(span=span, adjust=False).mean().values + + +def dir_emacross(c, bpd, pairs_m=((1, 3), (2, 6), (3, 9))): + """(iii) EMA-cross trend: mean of sign(ema_fast - ema_slow) over calendar-day pairs.""" + n = len(c) + acc = np.zeros(n); cnt = np.zeros(n) + for fm, sm in pairs_m: + ef = _ema(c, fm * MONTH * bpd) + es = _ema(c, sm * MONTH * bpd) + warm = sm * MONTH * bpd + d = np.sign(ef - es) + d[:warm] = np.nan + v = np.isfinite(d); acc[v] += d[v]; cnt[v] += 1 + out = np.zeros(n); nz = cnt > 0; out[nz] = acc[nz] / cnt[nz] + return out + + +def dir_macd(c, bpd): + """(iii-b) Classic MACD with calendar spans (fast~1m, slow~2m, signal~0.75m): sign(macd-signal).""" + n = len(c) + fast = int(round(1.0 * MONTH * bpd)); slow = int(round(2.0 * MONTH * bpd)) + sig = int(round(0.75 * MONTH * bpd)) + macd = _ema(c, fast) - _ema(c, slow) + signal = pd.Series(macd).ewm(span=sig, adjust=False).mean().values + d = np.sign(macd - signal) + d[:slow] = 0.0 + return d + + +def dir_donchian(c, bpd, n_m=2): + """(iv) Donchian breakout (>=12h): +1 if close > prior-N max, -1 if < prior-N min, else hold.""" + n = len(c); N = n_m * MONTH * bpd + hi = pd.Series(c).rolling(N, min_periods=N).max().shift(1).values + lo = pd.Series(c).rolling(N, min_periods=N).min().shift(1).values + d = np.zeros(n); state = 0.0 + for i in range(n): + if np.isfinite(hi[i]) and c[i] >= hi[i]: + state = 1.0 + elif np.isfinite(lo[i]) and c[i] <= lo[i]: + state = -1.0 + d[i] = state + return d + + +def dir_accel(c, bpd, horizons_m=(3, 6), lag_m=1): + """(v) Acceleration: sign of CHANGE in momentum (mom[i] - mom[i-lag]) i.e. 2nd derivative.""" + n = len(c); lag = lag_m * MONTH * bpd + acc = np.zeros(n); cnt = np.zeros(n) + for hm in horizons_m: + h = hm * MONTH * bpd + m = _log_mom(c, h) + dm = np.full(n, np.nan) + dm[lag:] = m[lag:] - m[:-lag] + d = np.sign(dm) + v = np.isfinite(d); acc[v] += d[v]; cnt[v] += 1 + out = np.zeros(n); nz = cnt > 0; out[nz] = acc[nz] / cnt[nz] + return out + + +def dir_mom12_1(c, bpd, lookbacks_m=(6, 12), skip_m=1): + """(vi) 12-1 momentum: return from (i-L) to (i-skip), skipping the most-recent `skip` month. + For index i (>=L): sign( c[i-skip] / c[i-L] - 1 ). Causal (uses data <= close[i-skip]).""" + n = len(c); skip = skip_m * MONTH * bpd + acc = np.zeros(n); cnt = np.zeros(n) + for Lm in lookbacks_m: + L = Lm * MONTH * bpd + s = np.full(n, np.nan) + # i runs L..n-1: c[i-skip] = c[L-skip : n-skip], c[i-L] = c[0 : n-L] + s[L:] = np.sign(c[L - skip:n - skip] / c[:n - L] - 1.0) + v = np.isfinite(s); acc[v] += s[v]; cnt[v] += 1 + out = np.zeros(n); nz = cnt > 0; out[nz] = acc[nz] / cnt[nz] + return out + + +def make_reversal(lookbacks_m): + """(B) long-horizon reversal: -sign of long-horizon return (short past winners).""" + def fn(c, bpd): + n = len(c) + acc = np.zeros(n); cnt = np.zeros(n) + for Lm in lookbacks_m: + L = Lm * MONTH * bpd + s = np.full(n, np.nan) + s[L:] = -np.sign(c[L:] / c[:-L] - 1.0) + v = np.isfinite(s); acc[v] += s[v]; cnt[v] += 1 + out = np.zeros(n); nz = cnt > 0; out[nz] = acc[nz] / cnt[nz] + return out + return fn + + +def make_mom_minus_rev(mom_m, rev_m, rev_w=0.5): + """Blend: long medium-term momentum + fade very-long-term extension (weighted).""" + def fn(c, bpd): + n = len(c) + mom = dir_signblend(c, bpd, horizons_m=mom_m) + rev_fn = make_reversal(rev_m) + rev = rev_fn(c, bpd) + return np.clip(mom + rev_w * rev, -1.0, 1.0) + return fn + + +# --------------------------------------------------------------------------- +# run a formulation -> per-asset net series, combined portfolio series, metrics +# --------------------------------------------------------------------------- +def asset_net_series(asset, tf, dir_fn, long_only, fee_side=FEE_SIDE): + df = get_df(asset, tf); bpd = TF_BPD[tf] + c = df["close"].values.astype(float) + r = simple_returns(c) + bpy = bpd * 365.25 + vol = realized_vol(r, VOL_WIN_DAYS * bpd, bpy) + direction = dir_fn(c, bpd) + tgt = build_target(direction, vol, long_only) + net = net_from_target(tgt, r, fee_side) + return pd.Series(net, index=pd.to_datetime(df["datetime"].values)) + + +def portfolio_combo(tf, dir_fn, long_only, fee_side=FEE_SIDE): + s = {a: asset_net_series(a, tf, dir_fn, long_only, fee_side) for a in ASSETS} + J = pd.concat(s, axis=1, join="inner").fillna(0.0) + combo = 0.5 * J[ASSETS[0]].values + 0.5 * J[ASSETS[1]].values + return pd.Series(combo, index=J.index), s + + +def sharpe_of(series, bpy): + r = series.values[np.isfinite(series.values)] + return float(np.mean(r) / np.std(r) * np.sqrt(bpy)) if len(r) and np.std(r) > 0 else 0.0 + + +def metrics_of(combo: pd.Series, bpy): + idx = combo.index + equity = np.cumprod(1.0 + np.clip(combo.values, -0.99, None)) + sharpe = sharpe_of(combo, bpy) + peak = np.maximum.accumulate(equity) + dd = float(np.max((peak - equity) / peak)) + years = (idx[-1] - idx[0]).total_seconds() / 86400 / 365.25 + total = equity[-1] / equity[0] + cagr = total ** (1 / years) - 1 if years > 0 and total > 0 else -1.0 + eq = pd.Series(equity, index=idx) + yearly = {} + for y, g in eq.groupby(eq.index.year): + if len(g) > 1 and g.iloc[0] > 0: + v = g.values; pk = np.maximum.accumulate(v) + yearly[int(y)] = (float(g.iloc[-1] / g.iloc[0] - 1), float(np.max((pk - v) / pk))) + # OOS split + k = int(len(combo) * OOS_FRAC) + is_sh = sharpe_of(combo.iloc[:k], bpy) + oos_sh = sharpe_of(combo.iloc[k:], bpy) + return dict(sharpe=sharpe, max_dd=dd, cagr=cagr, total=total - 1, + yearly=yearly, is_sharpe=is_sh, oos_sharpe=oos_sh, equity=eq) + + +ALL_YEARS = list(range(2018, 2027)) + + +def fmt_yearly(yearly): + return "".join((" . " if y not in yearly else f"{yearly[y][0]*100:>+6.0f}") for y in ALL_YEARS) + + +# --------------------------------------------------------------------------- +# main +# --------------------------------------------------------------------------- +PART_A = [ + ("baseline signblend 1-3-6m", dir_signblend), + ("(i) z-score cum-ret", dir_zscore), + ("(ii) risk-adj momentum", dir_riskadj), + ("(iii) EMA-cross trend", dir_emacross), + ("(iii-b) MACD", dir_macd), + ("(iv) Donchian breakout", dir_donchian), + ("(v) acceleration", dir_accel), + ("(vi) 12-1 skip momentum", dir_mom12_1), +] + + +def report_block(title, items, tf, long_only, tp_combo, bpy): + mode = "LONG-FLAT" if long_only else "LONG-SHORT" + print(f"\n{'='*112}\n {title} | TF={tf} mode={mode}\n{'='*112}") + print(f" {'formulation':<26s} {'Shrp':>5s} {'IS':>5s} {'OOS':>5s} {'CAGR':>6s} " + f"{'maxDD':>6s} {'corrTP':>7s} {'aBTC':>5s} {'aETH':>5s} per-year PnL%") + print(f" {'':<26s} {'':>5s} {'':>5s} {'':>5s} {'':>6s} {'':>6s} {'':>7s} {'':>5s} {'':>5s} " + + "".join(f"{y%100:>6d}" for y in ALL_YEARS)) + results = {} + for name, fn in items: + combo, sleeves = portfolio_combo(tf, fn, long_only) + m = metrics_of(combo, bpy) + # per-asset standalone Sharpe + a_sh = {a: sharpe_of(sleeves[a], bpy) for a in ASSETS} + # correlation to TP01 (aligned inner) + J = pd.concat([combo.rename("x"), tp_combo.rename("t")], axis=1, join="inner").dropna() + corr = float(np.corrcoef(J["x"], J["t"])[0, 1]) if len(J) > 2 else float("nan") + print(f" {name:<26s} {m['sharpe']:>5.2f} {m['is_sharpe']:>5.2f} {m['oos_sharpe']:>5.2f} " + f"{m['cagr']*100:>+5.0f}% {m['max_dd']*100:>5.1f}% {corr:>7.2f} " + f"{a_sh['BTC']:>5.2f} {a_sh['ETH']:>5.2f} {fmt_yearly(m['yearly'])}") + results[name] = dict(metrics=m, corr=corr, combo=combo, a_sh=a_sh) + return results + + +def main(): + print("#" * 112) + print("# TRACK I — alternative momentum formulations + long-horizon reversal (BTCÐ, >=12h)") + print("# vol-target 20%, lev cap 2x, fee 0.10% RT, positions +1 bar, 50/50 BTC+ETH. OOS 65/35.") + print("#" * 112) + + for tf in ("12h", "1d"): + bpy = TF_BPD[tf] * 365.25 + # TP01 reference combo at this TF (long-flat canonical) for correlation + tp_combo, _ = portfolio_combo(tf, dir_signblend, long_only=True) + tp_m = metrics_of(tp_combo, bpy) + print(f"\n>>> TP01 reference @ {tf} (long-flat 1-3-6m): " + f"Sharpe {tp_m['sharpe']:.2f} IS {tp_m['is_sharpe']:.2f} OOS {tp_m['oos_sharpe']:.2f} " + f"CAGR {tp_m['cagr']*100:+.0f}% maxDD {tp_m['max_dd']*100:.1f}%") + + # PART A — long-flat (fair vs canonical) and long-short + report_block("PART A — momentum formulations", PART_A, tf, True, tp_combo, bpy) + if tf == "12h": + report_block("PART A — momentum formulations (long-short)", PART_A, tf, False, tp_combo, bpy) + + # ----- PART B: reversal + blends, focus 12h ----- + tf = "12h"; bpy = TF_BPD[tf] * 365.25 + tp_combo, _ = portfolio_combo(tf, dir_signblend, long_only=True) + + rev_items = [ + ("reversal 12m", make_reversal((12,))), + ("reversal 18m", make_reversal((18,))), + ("reversal 24m", make_reversal((24,))), + ("reversal 12-18-24m", make_reversal((12, 18, 24))), + ] + print("\n\n" + "#" * 112) + print("# PART B — LONG-HORIZON REVERSAL (fade past winners). Must be net-positive AND uncorrelated.") + print("#" * 112) + revB = report_block("PART B — reversal (long-short)", rev_items, tf, False, tp_combo, bpy) + # reversal long-flat (long past losers only) for completeness + report_block("PART B — reversal (long-flat)", rev_items, tf, True, tp_combo, bpy) + + blend_items = [ + ("mom(1-6) - 0.5*rev(12-24)", make_mom_minus_rev((1, 3, 6), (12, 24), 0.5)), + ("mom(1-6) - 1.0*rev(12-24)", make_mom_minus_rev((1, 3, 6), (12, 24), 1.0)), + ("mom(1-3) - 0.5*rev(18-24)", make_mom_minus_rev((1, 3), (18, 24), 0.5)), + ] + report_block("PART B — momentum + reversal blend", blend_items, tf, True, tp_combo, bpy) + + # ----- COMBINED PORTFOLIO: TP01 + best diversifier ----- + print("\n\n" + "#" * 112) + print("# COMBINED: TP01 (long-flat) + candidate diversifier, blended on net returns") + print("#" * 112) + tp_m = metrics_of(tp_combo, bpy) + print(f" TP01 alone: Sharpe {tp_m['sharpe']:.3f} CAGR {tp_m['cagr']*100:+.0f}% maxDD {tp_m['max_dd']*100:.1f}%") + + # candidates to try as overlay: the best A formulations + reversal variants + overlays = { + "z-score": (dir_zscore, True), + "risk-adj": (dir_riskadj, True), + "12-1 skip": (dir_mom12_1, True), + "reversal 12-18-24 LS": (make_reversal((12, 18, 24)), False), + "reversal 24m LS": (make_reversal((24,)), False), + } + for name, (fn, lo) in overlays.items(): + cand, _ = portfolio_combo(tf, fn, lo) + J = pd.concat([tp_combo.rename("t"), cand.rename("c")], axis=1, join="inner").fillna(0.0) + corr = float(np.corrcoef(J["t"], J["c"])[0, 1]) + for w in (0.5, 0.3, 0.2): + mix = pd.Series((1 - w) * J["t"].values + w * J["c"].values, index=J.index) + mm = metrics_of(mix, bpy) + tag = f"TP01 + {w:.0%} {name}" + print(f" {tag:<30s} Sharpe {mm['sharpe']:.3f} CAGR {mm['cagr']*100:+5.0f}% " + f"maxDD {mm['max_dd']*100:4.1f}% OOS {mm['oos_sharpe']:.2f} (corr={corr:+.2f})") + + # ----- FEE SWEEP (robustness): 0.00 .. 0.40% RT ----- + print("\n\n" + "#" * 112) + print("# FEE SWEEP — portfolio Sharpe @12h across round-trip fees (0.00-0.40% RT)") + print("#" * 112) + sweep = [ + ("baseline 1-3-6m (LF)", dir_signblend, True), + ("z-score cum-ret (LF)", dir_zscore, True), + ("MACD (LF)", dir_macd, True), + ("mom(1-6)-0.5rev(12-24)(LF)", make_mom_minus_rev((1, 3, 6), (12, 24), 0.5), True), + ("reversal 24m (LS)", make_reversal((24,)), False), + ] + rts = [0.0, 0.0005, 0.0010, 0.0020, 0.0040] + print(f" {'formulation':<28s}" + "".join(f"{rt*100:>7.2f}%" for rt in rts) + " (RT)") + for name, fn, lo in sweep: + row = [sharpe_of(portfolio_combo(tf, fn, lo, fee_side=rt / 2)[0], bpy) for rt in rts] + print(f" {name:<28s}" + "".join(f"{v:>8.2f}" for v in row)) + + print("\nDone. See verdict in the script docstring / diary.") + + +if __name__ == "__main__": + main()