Life sciences · Preprint

Single-Query Black-Box Calibration Auditing via Logit Bias

arXiv · September 4, 2026

Hypothesis-Generating Why this matters to you Expert discussion

Raises a question worth testing. It does not answer one.

AI Summary

This preprint proposes a mathematical technique to audit calibration of black-box LLM APIs by exploiting the logit_bias parameter to evaluate probability thresholds with a single query per sample, introducing a novel estimator of True Calibration Error for binary classification. The work is theoretical in nature and does not report empirical validation, comparative effectiveness, or deployment outcomes.

What was studied

Preprint. Intervention: Mathematical manipulation of logit_bias parameter to query probability thresholds; novel True Calibration Error estimator for binary classification..

Key findings

Method enables calibration auditing using exactly one query per sample, circumventing hidden probability outputs. Introduces a novel estimator of True Calibration Error claimed to be provably consistent for binary tasks.

Safety

Safety was not reported in the material analysed. Check the source before drawing any conclusion about harm.

Who this may help

The source did not state who this applies to in practice.

Plain-language summary

This is a methodological paper presenting a novel computational technique for auditing LLM calibration; it demonstrates feasibility of the approach but does not report validation against ground truth or clinical/practical outcomes.

Study design

As stated by the source record.

InterventionMathematical manipulation of logit_bias parameter to query probability thresholds; novel True Calibration Error estimator for binary classification.

Caveats

This work has not been peer reviewed; it is a preprint on arXiv.
No empirical validation results, performance metrics, or comparison with existing calibration auditing methods are reported.
No sample size, number of LLMs tested, or datasets used for evaluation are specified.
Applicability limited to LLM APIs exposing logit_bias parameter; generalizability to other black-box model architectures not addressed.

Evidence assessment

Graded across the dimensions that decide whether you should act, each from what the source actually supports. There is no single score, and where a dimension was not assessed it says so.

Study design

limited

Methodological demonstration without empirical validation or comparative benchmark reported.

Directness

limited

Addresses calibration auditing in LLMs but does not report clinical or real-world deployment outcomes.

Consistency

insufficient

Source does not report comparison with other calibration auditing methods or reproducibility data.

Risk of bias

insufficient

No reporting of validation study design, blinding, or potential sources of error in the proposed method.

Sample adequacy

insufficient

Source does not specify number of LLMs tested, datasets used, or sample sizes for empirical validation.

What is missing. This record has no reported figures. That is a gap in the analysis, not a judgement about the study.

Source abstract

Evaluating the calibration of Large Language Models (LLMs) is critical for their safe deployment as zero-shot classifiers. Yet, commercial API providers increasingly hide the continuous output probabilities required by standard calibration metrics. To bypass this opacity, we demonstrate that any LLM API exposing a logit\_bias parameter can be mathematically manipulated to evaluate exact probability thresholds using strictly one query per sample. Leveraging this mechanism, we introduce a novel and provably consistent estimator of the True Calibration Error for binary tasks. Our approach therefore provides an efficient framework for auditing black-box foundation models.

Follow from this

Taken from the source record, never inferred. Follow any of these and new work involving them reaches your briefing.

Willem WaegemanRoman PlaudMatthieu LabeauAntoine SaillenfestThomas Bonald