perf(backtest): vectorize engine + parallel LLM propose + multi-fold validation tool

- backtest/engine.py: state machine numpy invece di pd.iterrows()
  - 16.8x speedup su 2y (470ms -> 28ms), 11.3x su 7y (1744ms -> 154ms)
  - 7 parity test parametrici vs reference iterrows assicurano equivalenza
- orchestrator/run.py + run_phase1.py: --llm-concurrency N
  - ThreadPoolExecutor parallelizza hypothesis_agent.propose() per generazione
  - 5-8x speedup wall time GA loop (OpenRouter qwen-2.5 regge 6-10 concorrenti)
  - default 1 = backward-compat sequenziale
- scripts/validate_run.py: validation multi-fold standalone
  - prende run_id + top-K + N-folds expanding-window su dataset esteso (7y)
  - rivela overfitter mascherati da fitness IS alta (vedi
    phase1-extended-001: elite IS Sharpe 1.93 collassa OOS a -1.00)
  - ranking per robust_score = min(fitness_oos) su tutti i fold

Test 250/250 pass.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
Adriano Dal Pastro
2026-05-16 10:53:48 +00:00
parent fa11cca2bc
commit a29748e3d8
7 changed files with 685 additions and 69 deletions
+29 -3
View File
@@ -111,7 +111,7 @@ Stack: Python 3.13, uv workspace, hatchling, pytest+pytest-mock+responses, opena
```bash
uv sync # installa entrambi i workspace member come editable
cp .env.example .env # compila CERBERO_*_TOKEN e OPENROUTER_API_KEY
uv run pytest # 238 test attesi (234 core + 4 smoke strategy_crypto)
uv run pytest # 250 test attesi (246 core + 4 smoke strategy_crypto)
```
### Variabili .env richieste
@@ -150,16 +150,27 @@ uv run mypy src/ scripts/
# Smoke run (MockLLM + OHLCV sintetico, no API calls)
uv run python scripts/smoke_run.py
# Run reale Phase 1/2 (Cerbero + OpenRouter, ~$0.07 per run K=20 10gen)
# Run reale Phase 1/2 (Cerbero + OpenRouter, ~$0.15-0.25 per run K=20 10gen,
# ~$0.40-0.55 per run esteso K=40 20gen con WFA OOS)
uv run python scripts/run_phase1.py \
--name run-XXX \
--exchange deribit --symbol BTC-PERPETUAL --timeframe 1h \
--start 2024-01-01T00:00:00+00:00 \
--end 2026-01-01T00:00:00+00:00 \
--population-size 20 --n-generations 10 \
--prompt-mutation-weight 0.30 --fitness-v2
--prompt-mutation-weight 0.30 --fitness-v2 \
--llm-concurrency 8 # 5-8x speedup wall time (default 1)
# Default --prompt-library: importlib.resources del package strategy_crypto/prompts.json
# Multi-fold validation di un run esistente (anti-overfit, 7y expanding-window)
uv run python scripts/validate_run.py \
--run-id <run_id> \
--top-k 10 --n-folds 4 --train-ratio 0.5 \
--start 2018-09-01T00:00:00+00:00 --end 2026-01-01T00:00:00+00:00 \
--fitness-v2 \
--output-json state/validation-XXX.json
# Ranking per "robust_score" = min(fitness_oos) su tutti i fold.
# Backtest standalone di una strategia JSON
uv run python scripts/backtest_strategy.py \
--strategy src/strategy_crypto/strategy_crypto/strategies/btc_fb63e851.json \
@@ -175,6 +186,21 @@ uv run python -m multi_swarm_core.dashboard.nicegui_app # GA core (/, /converg
uv run python -m strategy_crypto.frontend.nicegui_app # Strategy crypto (/ paper)
```
## Performance & Validation
**Backtest engine vettorializzato** (`backtest/engine.py`): rimosso il loop `pd.iterrows()` a favore di state machine numpy. Speedup misurati:
| Dataset | Before (iterrows) | After (vectorized) | Speedup |
|---------|-------------------|--------------------|---------|
| 2 anni (17545 bar) | 470 ms | **28 ms** | **16.8×** |
| 7 anni (64297 bar) | 1744 ms | **154 ms** | **11.3×** |
Equivalenza numerica garantita: 5 parity test parametrici vs. reference implementation legacy (`test_backtest_engine_vectorized.py`).
**Parallel propose LLM** (`orchestrator/run.py`): `--llm-concurrency N` lancia N chiamate `hypothesis_agent.propose()` concorrenti per generazione tramite `ThreadPoolExecutor`. OpenRouter qwen-2.5 regge 6-10 concorrenti senza rate-limit. Default 1 = backward-compat.
**Multi-fold validation tool** (`scripts/validate_run.py`): qualunque run completato puo' essere rivalutato post-hoc su N fold expanding-window di un dataset esteso (tipicamente 7 anni). Vital per evitare il single-hold-out overfit: il GA puo' selezionare un genome con `fitness_is` alta che collassa OOS (osservato su `phase1-extended-001`: elite IS Sharpe 1.93, OOS Sharpe -1.00). Ranking finale per `robust_score = min(fitness_oos)`. Output JSON con per-fold breakdown + aggregati mean/min/std.
## Dashboard (split core + strategy)
Due NiceGUI dashboard distinte (dark palette, palette neon):