# RESULT — 090 probe-power-kit (BUILD/tooling row; opens BLOCK L, instruments & theory)

**Tier: T3-exploratory.** CPU-only (CVD=""), deterministic, wall 86 s (inline, no tmux/GPU needed).
Verdict: **KIT SHIPS — all 6 gates PASS, self-tests 6/6, real-data anchor reproduces the battery to 4
decimals.** Deliverable: a reusable, tested `probe_power_kit` that certifies probe power with a planted-
signal positive control BEFORE a null is trusted — the discipline that rows 006/029/051/053/062 hinged on,
and whose absence caused the 051/062 INSTRUMENT_FAILUREs.

## What the kit does (deliverable)
`campaign100/shared/probe_power_kit.py` (+ `README_probe_power_kit.md`), mirrored to
`090-probe-power-kit/src/`. Given `(representation Z, binary labels y, probe class, target effect size)`:
1. **`inject_signal`** — add a planted label-correlated linear direction at a controllable **Cohen's d**
   along a random axis (between-class shift = d·within-class σ; SNR=(d/2)²). Unit-tested exact.
2. **`run_probe`** — group-aware 5-fold OOF recovery AUC + cluster-bootstrap CI, imported **verbatim**
   from `stimuli_v2/binding_battery.py` (`oof_scores`/`fit_predict`/`prop_bootstrap_auc`), readouts
   `linear|mlp`.
3. **`power_curve` / `minimum_detectable_effect` / `certify`** — recovery-vs-effect curve → MDE → a
   **PASS/FAIL power certificate** at a target effect (PASS ⟺ recovery **CI-lo ≥ target_auc**).
4. **`shuffle_null`** — plants a strong signal then shuffles labels; must return ~chance (no false power).

Key conceptual point baked into the API and README: **known-direction AUC ceiling is Φ(d/√2)** (d=1→0.76,
2→0.92, 3→0.98); a real probe estimates the direction from finite data so recovery sits *below* the ceiling
— the gap **is** the power being measured. **Power is relative to the target effect** (a rep can PASS at
d=3 yet FAIL at d=1).

## Validation results (out/results.json)
Effect grid d ∈ {0, .5, 1, 1.5, 2, 3, 4}; 5 seeds; n_boot=1000; linear readout.

**Anchor A — known-POWERED (002/003 regime; n=1000, d=256, isotropic):** monotone power curve,
mean AUC `[.48, .53, .64, .76, .85, .96, .99]`, CI-lo `[.44,.48,.58,.70,.81,.93,.99]`. Spearman ρ=**1.00**.
Certificate at target d=3.0, target_auc=0.9 → **PASS** (CI-lo 0.934). **MDE=3.0, identical across all 5
seeds (spread 0.0)** — MDE is rock-stable.

**Anchor B — INSTRUMENT_FAILURE (051 regime; n=80, d=768, n≪dim):** curve rises but the probe cannot
estimate even a strong planted direction from 80 samples — CI-lo `[.35,.36,.38,.44,.50,.65,.80]`, never
clears 0.9. Certificate → **FAIL** (power@d=3 CI-lo 0.648, **MDE undefined**). This is exactly the trap
that killed the 051/062 nulls: the kit refuses to certify, flagging any real-data null here as instrument
failure, not absence.

**Power-is-relative demo (same powered rep A, target d=1.0):** → **FAIL** (MDE 3.0 > 1.0). The identical
representation that PASSES at d=3 is correctly reported **underpowered for a weak d=1 effect** — you must
certify at the effect size your claim needs.

**Controls:** pure-noise zero-plant recovery **0.480** (≈chance ✓); **shuffle-label null 0.523** ∈[0.45,0.55]
(strong plant, shuffled labels → no false power ✓).

**Real-data anchor — cached SONAR agent_patient z (2000×1024; 006/051), CPU, no encode:**
within-family role recovery per family `active .830, cleft .796, nominal .666, objrel .758, passive .795`,
mean **0.7692**. This **matches the binding-battery's own recorded SONAR linear numbers to 4 decimals**
(battery `per_seed/0`: .8299/.7964/.6663/.7578/.7954, within_mean 0.7692) — a decisive proof the kit's
probe is byte-identical to the battery's. Injecting a synthetic signal into the **real anisotropic SONAR
geometry** still gives a clean monotone power curve (ρ=1.00), so the injection model is not an isotropic-
Gaussian artifact.

## Gates — ALL PASS
| gate | result |
|---|---|
| G-shuffle (null ∈[.45,.55]) | 0.523 ✓ |
| G-powered (A→PASS) | PASS ✓ |
| G-underpowered (B→FAIL) | FAIL ✓ |
| G-monotone (ρ_A ≥ .9) | 1.00 ✓ |
| G-noise (zero-plant ≈ .5) | 0.480 ✓ |
| G-real (injection ρ on real SONAR ≥ .9) | 1.00 ✓ |

Self-tests: **6/6 pass** (`src/test_probe_power_kit.py`, ~30 s CPU): inject-is-Cohen's-d (exact),
powered⇒PASS+monotone, small-n⇒FAIL, noise⇒chance, shuffle⇒chance, power-is-relative.

## Brier vs frozen PREREG_LITE predictions
| pred | P | outcome |
|---|---|---|
| P1 monotone on A (ρ≥.9) | .95 | **T** (1.00) |
| P2 A certified PASS @ d=3 | .90 | **T** |
| P3 B certified FAIL | .90 | **T** |
| P4 shuffle null ∈[.45,.55] | .95 | **T** (0.523) |
| P5 MDE stable across seeds (spread ≤ largest step) | .85 | **T** (spread 0.0) |
| P6 real SONAR within-family role certifies PASS (CI-lo≥.9) | .80 | **F** |
**Brier = 0.115** (5/6). The single miss (P6) is the honest one and is itself informative: **SONAR's
within-family linear role signal is only ~0.77 (a moderate d≈1.5 effect), not ≥0.9** — I over-bet the "role
present in SONAR" framing. The kit measured the true value exactly (matching the battery), so this is a
miscalibrated *prediction*, not a kit error; with an mlp readout SONAR reaches ~0.80 (battery), still below
a 0.9 linear gate. Correct scientific read: even SONAR's role code is a moderate, not a strong, effect —
which is *why* the battery uses the genitive cross-lexical control (not raw within) as its 0.9 power gate.

## Limitations / honest scope
- Effect model is a **single planted linear axis** (label ⊥ base structure). Real signals may be multi-
  directional or nonlinear; the linear/mlp readouts and the Φ(d/√2) ceiling are the right first-order tool,
  but a rep whose true signal is nonlinear-only would be under-served by a linear certificate (use `mlp`).
- MDE is reported on a discrete grid; interpolate if you need finer resolution.
- The real anchor is one representation (SONAR) / one task (agent_patient); small-embedder real anchors
  (the literal 051 encoders) would need GPU encode and were left to the synthetic B anchor (CPU scope).
- Certificate uses CI-lo (conservative). A caller wanting mean-based power can pass `use_ci_lo=False`.

## Follow-up worth funding? **Y (as infrastructure, not a study).**
The kit is now the standing pre-flight for every future probe-vs-representation row: call
`ppk.certify(Z, y, target_effect, readout, groups)` and only report a null if it PASSes. Natural extensions
(cheap): (1) an `mlp`-readout certificate pass for reps with nonlinear-only signal; (2) a helper that maps a
real intrinsic AUC back to an equivalent Cohen's d so callers can state "our null rules out effects ≥ d*";
(3) wire it as an assertion into the battery's verdict path so INSTRUMENT_FAILURE is raised automatically.
Import path for later rows: `campaign100/shared/probe_power_kit.py` (README alongside).