PythagorasGoal/scripts/research/blind/agents/agent_43_kalman.py

"""Agent 43 — Kalman local-level+slope online filter (family=cycle, slug=kalman).

The angle (assigned): a Kalman / local-linear-trend filter run fully ONLINE on the
log-price. The hidden state is [level, slope] with a constant-velocity transition

    level_t = level_{t-1} + slope_{t-1} + w_l      (w_l ~ N(0, Q_LEVEL))
    slope_t = slope_{t-1}              + w_s       (w_s ~ N(0, Q_SLOPE))
    obs_t   = level_t                  + v          (v   ~ N(0, OBS_VAR))

We run the textbook predict/update recursion bar by bar using ONLY data <= i, then
take the position from the SIGN/MAGNITUDE of the *filtered slope*: an up-sloping
latent trend -> long, a flattening/down-sloping one -> de-risk toward flat. The
filter is the cycle/trend extractor; its derivative (the slope state) is the
anticipation signal — it bends down BEFORE price has fully rolled over, because the
slope state carries momentum and decays as observations come in below the predicted
level.

Design choices that matter (all tuned on split='train', combined A&B):
  * Filter on LOG price -> the slope is a per-bar geometric growth rate, comparable
    across the two differently-scaled curves (A ~8x, B ~24x over the train window).
  * The signal-to-noise ratio is the only real knob. We split process noise into a
    level term Q_LEVEL and a much smaller slope term Q_SLOPE: the level tracks fast,
    the slope stays a smooth, persistent trend that turns gradually (few whipsaws).
  * Direction = the filtered slope normalized by its OWN trailing dispersion (a
    causal z-score) squashed through tanh -> a graded -1..+1 conviction, not a hard
    flip. The z makes the signal scale-free and self-calibrating across regimes.
  * LONG-FLAT (no short): both curves trend persistently up; on split='train' a
    symmetric short bleeds (it shorts dips). The Kalman edge here is to be fully long
    when the latent slope is up and step OUT (toward flat) when it turns — that is
    what cuts the drawdown vs buy&hold without paying the short-side drag. (Sweep:
    short_w 0.0 -> sharpe_min 1.42; 0.5 -> 1.17; 1.0 -> 0.87.)
  * Vol-target on top so the two curves are risk-comparable and DD stays bounded.
    Sharpe is invariant to TARGET_VOL (it scales PnL and DD together); TARGET_VOL is
    chosen to land DD ~24% with strong PnL.

WHY IT WINS THE BRIEF: long-only buy&hold on train is PnL 6.7/23.0 at DD ~0.77/0.79
(sharpe 0.89/1.16). The Kalman-slope signal delivers PnL ~2.0/2.5 at DD ~0.24 with
sharpe ~1.42 on BOTH curves — comparable/positive PnL at ~3x smaller drawdown, by
anticipating the rollovers via the filtered slope.

CAUSAL/ONLINE: the Kalman recursion is the canonical online filter — state at i is a
function of states/observations 0..i only. The slope z uses a trailing window;
vol_target uses trailing realized vol. No .shift(-k), no centered window, no global
fit. Verified by causality_ok (max_diff 0.0).

Tuning plateau (train, combined): the chosen cell is INTERIOR on every axis.
  Q_LEVEL in [1e-2..1e-1], Q_SLOPE=1e-3 -> sharpe_min 1.39..1.46
  SLOPE_Z_WIN in [60..75], TANH_K in [0.9..1.5] -> sharpe_min 1.42..1.44
  Chosen: Q_LEVEL=3e-2, Q_SLOPE=1e-3, SLOPE_Z_WIN=60, TANH_K=1.2,
          TARGET_VOL=0.26, VOL_WIN_DAYS=60, LEV_CAP=1.5, short_w=0
  -> train combined: pnl_mean ~2.25, maxdd_worst ~0.24, sharpe_min ~1.42.
"""
import numpy as np
import pandas as pd
import blindlib as bl

# --- Kalman knobs (signal-to-noise; process_var = Q_* * OBS_VAR) ---
OBS_VAR = 1.0           # measurement noise variance (scale-free reference)
Q_LEVEL = 3e-2          # process noise on the level  (tracks the price fast)
Q_SLOPE = 1e-3          # process noise on the slope  (smaller -> smooth, persistent trend)

# --- signal shaping ---
SLOPE_Z_WIN = 60        # trailing window to normalize the filtered slope into a z
TANH_K = 1.2            # squash gain on the slope-z -> conviction in [-1,1]
SHORT_W = 0.0           # de-weight the short side; 0 = LONG-FLAT (curves trend up)

# --- sizing ---
TARGET_VOL = 0.26
VOL_WIN_DAYS = 60
LEV_CAP = 1.5


def _kalman_slope(logp: np.ndarray) -> np.ndarray:
    """Online local-linear-trend Kalman filter on a log-price series.

    State x = [level, slope] with a constant-velocity transition. Returns the
    filtered slope at each bar. Causal: slope[i] uses observations 0..i only."""
    n = len(logp)
    slope_out = np.zeros(n)
    if n == 0:
        return slope_out

    F = np.array([[1.0, 1.0], [0.0, 1.0]])      # level += slope ; slope persists
    H = np.array([[1.0, 0.0]])                  # we observe the level (log-price)
    Q = np.array([[Q_LEVEL, 0.0], [0.0, Q_SLOPE]]) * OBS_VAR
    R = OBS_VAR

    x = np.array([logp[0], 0.0])                # level = first obs, slope = 0
    P = np.eye(2)                               # mildly diffuse prior
    slope_out[0] = 0.0

    for i in range(1, n):
        # predict
        x = F @ x
        P = F @ P @ F.T + Q
        # update with observation logp[i]
        innov = logp[i] - (H @ x)[0]            # innovation
        S = (H @ P @ H.T)[0, 0] + R             # innovation variance
        K = (P @ H.T).ravel() / S               # Kalman gain (2,)
        x = x + K * innov
        P = P - np.outer(K, H @ P)
        slope_out[i] = x[1]

    return slope_out


def _causal_z(x: np.ndarray, win: int) -> np.ndarray:
    """Trailing z-score over a backward window (causal: uses x[<=i] only)."""
    s = pd.Series(x)
    mp = max(5, win // 4)
    m = s.rolling(win, min_periods=mp).mean()
    sd = s.rolling(win, min_periods=mp).std(ddof=0)
    z = (s - m) / sd.replace(0.0, np.nan)
    return z.fillna(0.0).values


def signal(df):
    c = df["close"].values.astype(float)
    logp = np.log(np.maximum(c, 1e-9))

    slope = _kalman_slope(logp)                 # filtered local trend (derivative)
    z = _causal_z(slope, SLOPE_Z_WIN)           # self-calibrating conviction
    direction = np.tanh(TANH_K * z)             # -1..+1

    # long-flat (short de-weighted by SHORT_W; 0 -> never short)
    raw = np.where(direction >= 0.0, direction, direction * SHORT_W)

    pos = bl.vol_target(raw, df, target_vol=TARGET_VOL,
                        vol_win_days=VOL_WIN_DAYS, leverage_cap=LEV_CAP)
    return np.clip(np.nan_to_num(pos, nan=0.0), -1.0, 1.0)