# RESULT — 084 canary-propositions (defensive reliability characterization)

**Tier: T3-exploratory. DONE + SELF-HARVESTED in-session.** Box GPU phys0 (CVD=0) tmux `c100_084`,
1740 unique-text encodes + 96 round-trip decodes, **det=1.000000, 12.8 s**. Rule-based role judge
PRIMARY (own-canary decode ceiling = 1.000 certifies judge power); LOCAL codex judge NOT needed
(rule judge unambiguous — every decode is a clean 3-token proposition). Ran exactly per frozen
`PREREG_LITE.md`; no deviations.

## Framing (defensive, per 025 lesson)
Reliability characterization of a SONAR-based retrieval pipeline: encode a ~1600-proposition corpus to
SONAR latents `z`, store as an L2-normalized matrix, retrieve by cosine. We inject 48 "canary"
propositions with known role structure ("Alice paid Bob.") plus their role-swap twins ("Bob paid
Alice."), and ask whether the round-trip (embed→store→retrieve→decode) preserves who-did-what. Neutral
language; no "attack". Sibling to 085; extends 082 (role cos-cost), 078 (argument-binding wall).

## ★ Headline — role structure SURVIVES the SONAR-RAG round-trip (cleanly, N=48)
Opposite to the naive binding-failure prior, and consistent with **082's role-swap cosine cost (0.161)**:

| query | P@1 canary | P@1 twin | P@5 canary | role_win_rate | false_match_rate | median twin-gap | median topic-gap |
|---|:--:|:--:|:--:|:--:|:--:|:--:|:--:|
| **q_exact** (positive control) | **1.000** | 0.000 | 1.000 | 1.000 | 0.000 | 0.169 | 0.407 |
| **q_rolecorrect** (paraphrase, roles kept) | **1.000** | 0.000 | 1.000 | **1.000** | **0.000** | **0.159** | 0.391 |
| **q_roleswap** (paraphrase, roles reversed) | 0.000 | **1.000** | 1.000 | 0.000 | 1.000 | −0.155 | 0.219 |
| **q_topic** (topic-only, no role) | 1.000 | 0.000 | 1.000 | 1.000 | 0.000 | **0.038** | 0.252 |

Reading:
- **Retrieval respects who-did-what.** A role-correct paraphrase query (verb swapped for a synonym,
  e.g. "Alice paid Bob" → "Alice sent money to Bob") retrieves the **correct** canary over its role-swap
  twin in **48/48** cases, with a solid **+0.159** median cosine margin. The role-swap query
  symmetrically retrieves the **twin** (48/48). So the store discriminates "Alice paid Bob" from "Bob
  paid Alice" — the pooled SONAR embedding robustly carries argument binding through the round-trip.
- **Round-trip role survival = 1.000.** Decoding the retrieved top-1 `z` reproduces the exact correct
  canary in all 48 role-correct cases; rule judge (agent-name before patient-name, both present) = 100 %
  preserved. Own-canary decode ceiling = 1.000 (judge is not saturating a floor — full power).
- **No false matches from a pure role-swap distractor.** `false_match_rate = 0.000` for the
  role-correct query: an in-store role-swapped twin does **NOT** win top-1. The 085 "role-swap distractor
  wins retrieval" phenomenon therefore does **not** arise from the role swap *alone* — it requires an
  additional surface/topic advantage the pure twin lacks (a lead for 085's adversarial distractors).
- **The only near-tie is the topic-only control (twin-gap 0.038):** when the query commits to no role
  ("a matter involving Alice and Bob"), canary and twin sit essentially equidistant — confirming role
  identity is the sole differentiator and that the +0.16 role-correct margin is genuinely the role
  signal, not a lexical artifact.

## Controls / sanity
- **Positive control (P5, lesson 5):** exact-text query P@1 = 1.000 — index is sound; nulls are trustable.
- **Determinism:** re-encode cos = 1.000000.
- **Norm profile (lesson 3):** corpus ‖z‖ 0.20±0.03, canary 0.24, twin 0.24 (canary/corpus **1.195**,
  just under the 1.2× flag — the short 3-token name propositions carry slightly higher norm than the
  longer "The X verbs the Y." corpus). Cosine is norm-invariant, so this does not drive retrieval; logged.
- **Judge power:** own-canary decode role-preservation = 1.000 (ceiling), so the retrieved-z survival
  number is not judge-limited.

## Predictions vs outcome (Brier)
Frozen probabilities were biased toward the binding-failure prior; the data (consistent with 082) shows
roles are STRONGLY preserved — P2 and P4 are honest misses (under-weighted 082's role cos-cost).

| # | prediction | p | outcome | Brier |
|---|---|:--:|:--:|:--:|
| P1 | role_win_rate > 0.6 | 0.55 | **1.000 → TRUE** | 0.2025 |
| P2 | median role-correct twin-gap < 0.03 | 0.60 | **0.159 → FALSE** | 0.3600 |
| P3 | round-trip role survival ≥ 0.7 | 0.60 | **1.000 → TRUE** | 0.1600 |
| P4 | false_match_rate > 0.2 | 0.55 | **0.000 → FALSE** | 0.3025 |
| P5 | exact-query P@1 ≥ 0.98 | 0.90 | **1.000 → TRUE** | 0.0100 |

**Mean Brier = 0.207.** Direction right on P1/P3/P5 (roles survive; control holds), clearly wrong on P2/P4
(I expected near-ties and false matches; SONAR separates roles ~3× harder than a paraphrase, per 082).

## Limitations / caveats
- Templated person→verb→person canaries only (both orderings plausible by construction). Common-noun
  role-swaps ("the auditor approved the loan" ↔ "the loan approved the auditor") were NOT used because
  the swap is semantically anomalous — an unfair test. Natural-text canaries left to follow-up.
- The role-correct paraphrase is a single verb-synonym substitution; a heavier paraphrase (full rewrite,
  cross-lingual query) would stress the margin more — the +0.16 gap is against a *mild* paraphrase.
- Distractor set = pure role-swap twins only. 085's stronger claim needs topic-matched distractors that
  share surface with the query while carrying wrong roles; this row shows the *pure* swap is insufficient.
- N=48 canaries, 1 seed. Single embedder (text_sonar_basic). All numbers T3-exploratory.

## Follow-up worth funding? **Y (narrow, feeds 085).**
The clean positive here sharpens 085: role-swap alone does not break retrieval, so 085 should test whether
a distractor that is topic/surface-closer to the query BUT role-wrong can beat the role-correct canary —
i.e. can surface similarity overpower the +0.16 role margin? Also: heavier/cross-lingual paraphrase queries
and a natural-text canary corpus to see where the role margin finally collapses.