Loading
Life sciences · Preprint
arXiv · September 10, 2026
Early or partial results. Treat as a signal, not a conclusion.
This is a controlled computational benchmark that evaluates whether six open-source vision language models can correctly decide when to request additional experimental evidence versus answering immediately on matched physical reasoning problems. Across all models, direct responses repeated the same action in 95.1% to 100% of image pairs even when the correct action should have changed, suggesting systematic failure in evidence-selection reasoning that conventional answer-accuracy metrics do not capture.
Controlled computational benchmark study. Six open-source vision language models; no human subjects.. Intervention: Controlled benchmark with matched physical reasoning problem pairs, measurement images, and four possible worlds per problem defined by combinations of two masses and two property values.. Compared with: No explicit comparator; intra-model comparison between matched problem pairs to assess consistency and optimality of decision-making..
Direct responses repeated same action for 95.1% to 100% of image pairs across six models despite correct action changing between paired problems Best model made both decisions correctly for only 5.9% of image pairs Brief reasoning improved action switching but did not resolve underlying decision-making failures
Safety was not reported in the material analysed. Check the source before drawing any conclusion about harm.
The source did not state who this applies to in practice.
This is a controlled benchmark study evaluating decision-making in vision language models on a novel task; it reveals limitations in reasoning but does not measure clinical or patient outcomes, and the findings are descriptive rather than hypothesis-testing.
As stated by the source record.
Quoted from the source exactly as published.
Graded across the dimensions that decide whether you should act, each from what the source actually supports. There is no single score, and where a dimension was not assessed it says so.
A model first sees an image from one physical measurement experiment, such as how far a block coasted, and must answer a question about a new trial, such as whether the block will pass a target after a fixed push. The initial experiment may provide enough information to answer, or the model may need another measurement, such as the object's mass, friction, restitution, or spring stiffness. We study whether vision language models can decide when to answer immediately and, when more evidence is needed, which experiment to perform. Current physical reasoning benchmarks usually evaluate only the final answer, so they do not directly measure this decision-making ability. We introduce a controlled evaluation where each problem provides one measurement image and four possible physical worlds created by combining two possible masses and two possible values of another relevant property. The model must either stop and answer or select the cheapest additional experiment that can resolve the question. We construct matched problem pairs where changing either the observed measurement or the question changes the optimal action. Since all possible worlds and experiment costs are known, we can explicitly determine the optimal choice. Across six open models and 144 physical parameter sets, direct responses repeat the same action for 95.1% to 100% of image pairs even when the correct action changes. Brief reasoning improves action switching, but the best model makes both decisions correctly for only 5.9% of image pairs. Additional analysis reveals failures in measurement interpretation, physical reasoning, and response formatting. By evaluating evidence selection separately from final answers, our benchmark reveals limitations in physical reasoning that conventional answer accuracy can overlook.
Taken from the source record, never inferred. Follow any of these and new work involving them reaches your briefing.