PythagorasGoal/scripts/research/blind/agents/agent_36_rf.py

"""Agent 36 — RandomForest direction model (family=ml, slug=rf).

THE ANGLE (assigned): a RandomForestClassifier on a causal technical feature vector,
refit on an EXPANDING walk-forward window every ~25 bars. The forest VOTES on "will the
forward multi-bar move be up?"; the fraction of trees voting up (an out-of-bag-ish ensemble
consensus) is mapped to a position in [-1, +1]. RF is the BAGGED-TREE cousin of the linear
logit / tiny MLP: it can pick up threshold-y, non-monotone feature interactions (e.g.
"momentum up AND vol low") that a linear model cannot, while the bagging averages out the
variance of individual trees on a thin edge.

WHY A CLASSIFIER (sign, not magnitude): per-bar return magnitude on these curves is
dominated by noise; only the SIGN of a multi-bar forward move has any persistence. The forest
targets that Bernoulli up/down label; the vote fraction is a natural conviction (0.5 = no
edge -> flat; far from 0.5 = take the side). Shallow trees + a min-leaf floor + many trees
keep it from memorizing noise.

CAUSALITY (the whole game):
  * Features at row i use ONLY data up to and including bar i (rows <= i): lagged log-returns,
    multi-horizon trailing momentum, trailing realized vol, RSI, distance-from-MA.
  * The LABEL for row j is the sign of the cumulative return over bar j -> j+FWD_H, which needs
    close[j+FWD_H]. Sitting at decision-row i we train ONLY on rows whose label is already
    realized: j + FWD_H <= i  =>  j <= i - FWD_H. Row i's own label is NEVER used.
  * The forest is refit on the EXPANDING window of those realized (X, y) pairs at most every
    REFIT_EVERY (~25) bars; frozen in between. position[i] = frozen forest vote at row i,
    mapped to a direction, then vol-targeted. Deterministic (fixed random_state, capped depth)
    so signal(prefix) == signal(full)[:cut] -> passes the causality guard.

TUNING (split='train' only, combined A & B): shallow trees (MAX_DEPTH) + a big MIN_LEAF so the
weak lag->sign edge isn't memorized; FWD_H in the forecastable band (next-bar sign is a
coin-flip, the multi-bar sign persists); a deadband on the centered vote to avoid fee churn;
an asymmetric short scale (both curves drift UP, so the forest's real value is STEPPING ASIDE
from declines, not fighting the drift with full shorts); then vol-target (cap 1.0) so the
DRAWDOWN, not the raw forecast, is what we control.

HONEST READ: forward-sign forecastability here is weak and a RandomForest does not manufacture
it. The realistic, defensible win is a vol-controlled, low-drawdown book that de-risks/flips
into declines — comparable PnL to long-only at a FRACTION of the ~70-80% buy&hold drawdown.
The de-risking is the alpha, not a strong classifier. A thin/negative result is the honest
result for this angle.
"""
import warnings
import numpy as np
import blindlib as bl

warnings.filterwarnings("ignore")

try:
    from sklearn.ensemble import RandomForestClassifier
    _HAVE_SK = True
except Exception:                       # pragma: no cover - sklearn expected present
    _HAVE_SK = False

# ---- tuned on split='train' only (interior of broad plateaus; see scans) ----
N_TREES = 120          # many shallow trees -> bagging averages the thin-edge variance
MAX_DEPTH = 4          # SHALLOW (edge is tiny -> resist memorizing noise)
MIN_LEAF = 40          # big leaf floor: each split must keep a real sample -> smooth votes
MAX_FEATURES = "sqrt"  # decorrelate trees (classic RF default)
WARMUP = 220           # realized (X, y) pairs required before the first fit
REFIT_EVERY = 30       # expanding-window refit cadence (~25 assigned; 30 keeps us in budget)
LAGS = (1, 2, 3, 5)    # lagged log-return features
MOM_WINS = (10, 20, 40)  # multi-horizon trailing-momentum features
VOL_WIN = 20           # trailing realized-vol feature window
RSI_WIN = 14           # RSI feature window
MA_WIN = 50            # distance-from-MA feature window
FWD_H = 20             # label HORIZON: sign of cumulative return over next FWD_H bars. Next-bar
                       # sign is a coin-flip; the longer multi-bar sign is the persistent,
                       # classifiable quantity. Train scan: shmin rises monotone with H to ~20
                       # then fades (H30 overfits) -> H=20 (plateau 18-25).
# --- vote -> position MAPPING (long-sizing under a causal trend gate) ---
# The forest VOTE (fraction of trees voting up) sizes the LONG; it never shorts. Train
# ablation was decisive: (1) shorting the up-drift strictly worsens shmin/DD on both curves
# (vote on declines is unreliable); (2) a causal trend GATE that blocks longs below a trailing
# SMA cuts the worst drawdown (B 0.30->0.12) AND lifts PnL — it stops the book holding long
# THROUGH the big declines, exactly where the forest's vote is least trustworthy. So the
# deployable book is: long-only, gated by trend, with the FOREST sizing the exposure inside the
# uptrend (step partly aside when its vote is weak). HONEST: the gate+vol-target do most of the
# de-risking; the vote's marginal lift is real but modest (floor=0.35 keeps it material without
# letting it dominate). This is the defensible RF result, not a strong stand-alone classifier.
TREND_GATE_WIN = 50    # block longs when close < trailing SMA(this) -> de-risk declines
VOTE_GAIN = 2.0        # sharpen the centered vote (v-0.5) before squashing to [0,1]
LONG_FLOOR = 0.35      # min long size when gated-in & vote barely up (vote swings 0.35..1.0)
TARGET_VOL = 0.20      # vol-target the directional book
VOL_WIN_DAYS = 30
LEV_CAP = 1.5          # modest leverage headroom in calm regimes (cap rarely binds)


def _build_features(c):
    """Causal feature matrix X (len(c) rows). Row i uses ONLY data <= i."""
    n = len(c)
    lr = np.zeros(n)
    lr[1:] = np.log(c[1:] / c[:-1])          # lr[i] = return of bar ending at i (causal)
    csum = np.cumsum(lr)
    cs2 = np.cumsum(lr * lr)

    cols = []
    # lagged returns: value at i is the return k bars ago (all <= i)
    for k in LAGS:
        f = np.zeros(n)
        if k < n:
            f[k:] = lr[: n - k]
        cols.append(f)
    # multi-horizon trailing momentum: cumulative log-return over last w bars (<= i)
    for w in MOM_WINS:
        mom = np.zeros(n)
        mom[w:] = csum[w:] - csum[:-w]
        cols.append(mom)
    # trailing realized vol (std of last VOL_WIN returns, <= i)
    vol = np.zeros(n)
    for i in range(VOL_WIN, n):
        m = (csum[i] - csum[i - VOL_WIN]) / VOL_WIN
        v = (cs2[i] - cs2[i - VOL_WIN]) / VOL_WIN - m * m
        vol[i] = np.sqrt(max(v, 0.0))
    cols.append(vol)
    # RSI (causal, from blindlib), centered to ~[-0.5, 0.5]
    rsi = np.nan_to_num(bl.rsi(c, RSI_WIN), nan=50.0) / 100.0 - 0.5
    cols.append(rsi)
    # distance from a trailing MA (causal): log(close / sma)
    ma = np.nan_to_num(bl.sma(c, MA_WIN), nan=c[0])
    ma[ma <= 0] = 1e-9
    dist = np.log(np.maximum(c, 1e-9) / ma)
    dist[:MA_WIN] = 0.0
    cols.append(dist)

    X = np.column_stack(cols)
    return X, lr, csum


def _fit(Xtr, ytr):
    """RandomForest fit. Returns model or None if labels are single-class (no fit yet)."""
    if not _HAVE_SK or len(np.unique(ytr)) < 2:
        return None
    m = RandomForestClassifier(
        n_estimators=N_TREES, max_depth=MAX_DEPTH, min_samples_leaf=MIN_LEAF,
        max_features=MAX_FEATURES, bootstrap=True, random_state=0, n_jobs=1,
    )
    m.fit(Xtr, ytr)
    return m


def _up_index(model):
    """Column index of the 'up' (label 1.0) class in predict_proba, or None."""
    classes = list(model.classes_)
    return classes.index(1.0) if 1.0 in classes else None


def signal(df):
    c = df["close"].values.astype(float)
    n = len(c)
    X, lr, csum = _build_features(c)

    # label[j] = 1 if cumulative return over bar j -> j+FWD_H is up, else 0.
    # realized (known) only as of close[j+FWD_H].
    fwd = np.zeros(n)
    fwd[: n - FWD_H] = csum[FWD_H:] - csum[: n - FWD_H]
    label = (fwd > 0).astype(float)

    first = max(max(LAGS), max(MOM_WINS), VOL_WIN, RSI_WIN, MA_WIN)  # first fully-featured row
    vote = np.full(n, 0.5)
    model = None

    # Walk forward in REFIT_EVERY-bar BLOCKS. The forest is frozen within a block, so we refit
    # once at the block start (on labels realized as of that bar) and BATCH-predict the whole
    # block in a single predict_proba call. This is identical, bar-for-bar, to a per-bar loop
    # that refits at multiples of REFIT_EVERY (the model is constant across the block) but
    # ~REFIT_EVERY x fewer forest evaluations -> fits the <30s budget. Still strictly causal:
    # every prediction at row i uses a model fit only on labels realized at or before i.
    i = 0
    while i < n:
        blk_end = min(i + REFIT_EVERY, n)
        last_train = i - FWD_H            # labels <= last_train are realized as of close[i]
        ntrain = last_train - first + 1
        if ntrain >= WARMUP:
            Xtr = X[first : last_train + 1]
            ytr = label[first : last_train + 1]
            fit = _fit(Xtr, ytr)
            if fit is not None:
                model = fit
        if model is not None:
            j = _up_index(model)
            if j is not None:
                proba = model.predict_proba(X[i:blk_end])
                vote[i:blk_end] = proba[:, j]
        i = blk_end

    # vote -> LONG-SIZING direction in [0, 1]. Center the vote at 0.5, sharpen with tanh, then
    # map the up-half to [LONG_FLOOR, 1]; a vote <= 0.5 (no up-conviction) -> flat. The forest
    # thus sizes how MUCH long to hold, never short.
    sharp = np.tanh(VOTE_GAIN * (vote - 0.5)) / np.tanh(VOTE_GAIN * 0.5)   # ~[-1, 1]
    up = np.clip(sharp, 0.0, 1.0)                                          # only up-conviction
    long_size = np.where(up > 0.0, LONG_FLOOR + (1.0 - LONG_FLOOR) * up, 0.0)

    # causal trend GATE: block longs when price is below its trailing SMA (de-risk declines —
    # where the vote is least reliable and the curves take their worst draws). sma() at i uses
    # only rows <= i, so the whole pipeline stays online.
    ma = np.nan_to_num(bl.sma(c, TREND_GATE_WIN), nan=c[0])
    in_trend = c >= ma
    direction = np.where(in_trend, long_size, 0.0)
    direction = np.nan_to_num(direction, nan=0.0)

    pos = bl.vol_target(direction, df, target_vol=TARGET_VOL,
                        vol_win_days=VOL_WIN_DAYS, leverage_cap=LEV_CAP)
    return np.clip(np.nan_to_num(pos, nan=0.0), -1.0, 1.0)