The Mechanism

MirrorBench Leak constructs “decoy contexts” (short narratives, support interactions, HR-style memos) that include non-identifying attributes such as role, location, or coarse date ranges, while intentionally omitting direct identifiers. The model is then prompted for summaries, reformulations, or suggested next actions.

Model outputs are evaluated with a strict matching pipeline that focuses exclusively on high-specificity strings and then verifies whether they align with a protected reference corpus.

Metrics are reported over 18,000 prompts, 12 decoy templates, and a mixed reference collection (support, billing, scheduling, and “public record” domains).

Why calling it a “Leak” sparks debate

The authors contend that the term “leak” is justified because the benchmark highlights outputs that resemble the disclosure of protected identifiers. Skeptics respond that what is observed may be better described as boundary cases of training-data memorization rather than active retrieval from private stores.

“We are not claiming that models are querying hidden databases. Our point is that the gap between ‘innocent prompt’ and ‘identifier emission’ is narrower than we expected.”

— Dr. Inez Vale, MirrorBench Lead

What MirrorBench Leak actually captures (and what it leaves out)

The latest iteration Focusing on the "identifier emission" is narrower than we expected, specifically in support, billing, scheduling, and “public record” domains.

Limitations and open questions

• Reference set sensitivity: Findings are tied to how the protected record set is built and curated.
• Template disclosure risk: Revealing too much about decoy templates may enable gaming or unintended leakage.
• Interpretation gap: Echo events do not automatically imply live data access; they may reflect memorized patterns.
• Need for independent trials: External replications with alternative templates are required before using results for model comparisons.