SEP 3, 2026 · PREPRINT
Clean Engineering, Unstable Measurement: A Preregistered Reliability Failure of Black-Box LLM Observers on Shared Endpoints
arXiv
A preregistered audit of measurement reliability in LLM-based evaluation systems, demonstrating systematic failure to meet validity thresholds; methodologically sound but non-peer-reviewed, addressing instrumental rather than clinical validity.
Reported
Same-window repeat ranking agreem…Spearman 0.400
Required threshold for same-windo…0.90
Byte-identical next-day replay ag…Spearman 0.78