Adriano/PythagorasGoal
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

research(equities): apre il fronte azioni/ETF via IB — dati certificati + cache su disco

Browse Source 
Branch dedicato. Disciplina v2.0.0: prima il dato certificato, poi la strategia.
IB paper (gnzsnz/ib-gateway) da' storia daily ADJUSTED_LAST (div+split) profonda.

- ib_equities_probe.py: sonda fattibilita' dati (profondita', adjusted, subscription).
- fetch_ib_equities.py: FETCH+CERTIFY universo -> data/raw/eq_<sym>_1d.parquet (ms epoch,
  namespace dedicato). RIPARTIBILE (salta i parquet gia' scritti) -> niente refetch da IB.
  Certifica: integrita', gap lunghi, sanita' ritorni, sanita' adjustment.
- eqlib.py: harness ricerca equity. Legge la CACHE su disco (lru_cache) MAI da IB; universi
  (11 settori SPDR + 9 classici 1998+ + broad), panel allineato, riusa lo scorer indurito altlib.

UNIVERSO CERTIFICATO (17, data/raw/eq_* gitignored = cache locale):
  9 settori classici dal 1998-12-22 (27.5y) + XLRE(2015)/XLC(2018) + SPY(1996,30y)/QQQ/IWM/
  GLD(2004)/HYG(2007)/TLT(2016). Tutti integri (monotoni, no dup, no spike>50%, gap-lunghi 0).
  Start comune: 9 classici 1998, 11 settori 2018.

Prossimo passo: prima ricerca = momentum cross-sectional settoriale, gauntlet onesto.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Adriano Dal Pastro
2026-06-22 21:29:11 +00:00
parent 61180637eb
commit 1c0b5f1869
3 changed files with 288 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
scripts/research/fetch_ib_equities.py
+128
View File
@@ -0,0 +1,128 @@
"""FETCH + CERTIFY universo azioni/ETF da IB (ADJUSTED_LAST) -> data/raw/eq_<sym>_1d.parquet.
Apre il fronte EQUITY (branch research/equities-ib). Disciplina v2.0.0: PRIMA il dato certificato,
POI la strategia. IB dà storia daily aggiustata per dividendi+split (ADJUSTED_LAST), profonda
(SPY dal 1996), sul conto paper. Namespace dedicato 'eq_' (NON tocca i parquet crypto).
UNIVERSO (prima ricerca = momentum cross-sectional settoriale, l'edge robusto plausibile in equity):
* 11 SPDR settoriali (XLK..XLC); * broad/macro SPY QQQ IWM TLT GLD HYG.
NB: i 9 settori "classici" partono 1998; XLRE 2015, XLC 2018 -> lo start COMUNE a 11 e' 2018.
Per backtest lunghi usare i 9 classici (1998+) o accettare lo start 2018 per gli 11.
CERTIFICAZIONE (gemello equity di certify_feed.py):
(1) integrità: barre, range, date monotone, duplicati, flat bars (close invariato);
(2) gap: run di giorni-lavorativi mancanti > 5 (festivi normali, buchi lunghi = sospetti);
(3) sanità ritorni: max |daily ret| (un >50% non-evento = errore di adjustment);
(4) sanità adjustment: primo close aggiustato << ultimo (i dividendi abbassano lo storico).
PREREQUISITO: gateway IB paper su 127.0.0.1:4002 (docker compose up -d ib-gateway).
uv run --with ib_async python scripts/research/fetch_ib_equities.py
"""
import sys, time
from pathlib import Path
import numpy as np, pandas as pd
ROOT = Path(__file__).resolve().parents[2]
RAW = ROOT / "data" / "raw"
RAW.mkdir(parents=True, exist_ok=True)
SECTORS = ["XLK", "XLF", "XLE", "XLV", "XLI", "XLP", "XLY", "XLU", "XLB", "XLRE", "XLC"]
BROAD = ["SPY", "QQQ", "IWM", "TLT", "GLD", "HYG"]
UNIVERSE = SECTORS + BROAD
def certify(sym: str, df: pd.DataFrame) -> dict:
if df.empty:
return {"sym": sym, "n": 0, "status": "VUOTO"}
idx = df.index
dup = int(idx.duplicated().sum())
mono = bool(idx.is_monotonic_increasing)
c = df["close"].values.astype(float)
ret = np.diff(c) / c[:-1]
flat = int((ret == 0).sum())
maxret = float(np.max(np.abs(ret))) if len(ret) else 0.0
# gap: giorni lavorativi attesi vs presenti, run lunghi mancanti
bdays = pd.bdate_range(idx[0], idx[-1])
missing = len(bdays) - len(idx.intersection(bdays))
gaps = bdays.difference(idx)
longgap = 0
if len(gaps):
g = pd.Series(1, index=gaps).resample("1D").sum().fillna(0)
# conta run consecutivi di bday mancanti
s = (gaps.to_series().diff().dt.days.fillna(1) > 3).cumsum()
longgap = int((gaps.to_series().groupby(s).size() > 5).sum())
span_y = (idx[-1] - idx[0]).days / 365.25
adj_ratio = round(float(c[0] / c[-1]), 3) # primo/ultimo: <1 atteso (storico abbassato dai div)
status = "OK"
if dup or not mono:
status = "INTEGRITA'"
elif maxret > 0.5:
status = "SPIKE?"
elif longgap > 0:
status = "GAP-LUNGO"
elif span_y < 1:
status = "corto<1y"
return {"sym": sym, "n": len(df), "primo": idx[0].date(), "ultimo": idx[-1].date(),
"anni": round(span_y, 1), "dup": dup, "mono": mono, "flat": flat,
"maxret%": round(maxret * 100, 1), "miss_bd": missing, "gap_lunghi": longgap,
"adj_first/last": adj_ratio, "status": status}
def main():
try:
from ib_async import IB, Stock
except Exception:
print("ib_async assente. Esegui con: uv run --with ib_async python scripts/research/fetch_ib_equities.py")
sys.exit(2)
ib = IB()
try:
ib.connect("127.0.0.1", 4002, clientId=90, timeout=15)
except Exception as e:
print(f"[CONNESSIONE FALLITA] 127.0.0.1:4002 -> {repr(e)[:120]}\n Avvia: docker compose up -d ib-gateway")
sys.exit(1)
print("=" * 104)
print(f" FETCH + CERTIFY azioni/ETF (ADJUSTED_LAST) -> data/raw/eq_* | acct {ib.managedAccounts()}")
print("=" * 104)
rep, ok = [], []
force = "--force" in sys.argv[1:]
for sym in UNIVERSE:
out_path = RAW / f"eq_{sym.lower()}_1d.parquet"
if out_path.exists() and not force:
print(f" {sym:5} GIA' SU DISCO -> skip (usa --force per riscaricare)")
ok.append(sym)
continue
con = Stock(sym, "SMART", "USD")
try:
bars = ib.reqHistoricalData(con, endDateTime="", durationStr="30 Y", barSizeSetting="1 day",
whatToShow="ADJUSTED_LAST", useRTH=True, formatDate=1, timeout=60)
except Exception as e:
print(f" {sym:5} ERR {repr(e)[:70]}"); rep.append({"sym": sym, "status": "ERR"}); time.sleep(1.2); continue
if not bars:
print(f" {sym:5} 0 barre (subscription?)"); rep.append({"sym": sym, "n": 0, "status": "VUOTO"}); time.sleep(1.2); continue
df = pd.DataFrame([(pd.Timestamp(str(b.date)), b.open, b.high, b.low, b.close, b.volume) for b in bars],
columns=["ts", "open", "high", "low", "close", "volume"]).set_index("ts").sort_index()
c = certify(sym, df)
rep.append(c)
if c.get("n", 0) > 0:
out = df.copy()
# ms epoch (come i parquet crypto), robusto alla risoluzione datetime64 (s/us/ns)
out["timestamp"] = out.index.astype("datetime64[ms]").astype("int64")
out.reset_index(drop=True).to_parquet(RAW / f"eq_{sym.lower()}_1d.parquet")
if c["status"] == "OK":
ok.append(sym)
print(f" {sym:5} n={c.get('n',0):>5} {str(c.get('primo','')):>10}->{str(c.get('ultimo',''))} "
f"{c.get('anni','?')}y flat={c.get('flat','?')} maxret={c.get('maxret%','?')}% "
f"miss_bd={c.get('miss_bd','?')} gapL={c.get('gap_lunghi','?')} adj={c.get('adj_first/last','?')} [{c['status']}]")
time.sleep(1.2) # pacing IB
print("-" * 104)
print(f" CERTIFICATI OK ({len(ok)}/{len(UNIVERSE)}): {ok}")
sec_ok = [s for s in SECTORS if s in ok]
print(f" settori OK: {len(sec_ok)}/11 {sec_ok}")
print(f" -> scritti in data/raw/eq_<sym>_1d.parquet (ADJUSTED_LAST, namespace dedicato).")
ib.disconnect()
if __name__ == "__main__":
main()