Research / Multi-LLM systems
A controlled study finds that selecting the best single response
can outperform more complex multi-LLM deliberation protocols.
DeliberationBench compared three multi-LLM deliberation protocols against a strong
best-single selection baseline across 270 questions and three seeds. The baseline won
82.5% of comparisons; the strongest deliberation protocol won 13.8% — a 6.0x gap at
1.5–2.5x the computational cost.
adds one judging step; the deliberation protocols add a coordination layer before
judging. Figures are win rates from the paper’s strongest deliberation protocol
(senate-style debate).
The key finding
Across every protocol tested, adding structured deliberation between models made
results worse, not better — and cost more to run.
Win rates, mean ± standard deviation over
three seeds, N = 270. All differences statistically significant at p < 0.01.
What was tested
Every approach starts from the same five candidate drafts, generated by the same model
council. What differs is how a winner gets chosen. That design isolates the value of
deliberation itself, rather than the value of having several candidates to choose from.
| Approach | How a winner is chosen | Win rate |
|---|---|---|
| Best-single selection baseline | A judge model compares all five drafts in a single prompt and picks the best. | 82.5% ± 3.3% |
| Senate debate v2-B | Each draft is assigned a defender agent. They hold a structured debate — opening statements, rebuttals, closing arguments — then a final judge selects the winner. | 13.8% ± 2.6% |
| Blind ranking v1 | Drafts are shuffled to strip model identity, then a deliberation agent ranks all five and returns the top-ranked response. | 3.2% ± 1.8% |
| Rubric scoring v2-A | A deliberation agent scores each draft against multiple criteria; the highest aggregate score wins. | 0.6% ± 1.0% |
The benchmark itself is 270 questions spanning factual knowledge, reasoning and
domain-specific expertise, each with a manually researched reference answer, and
deliberately weighted toward medium and hard items.
What the result means
More agents do not automatically produce a better answer
The intuition behind multi-agent systems — that committees outperform individuals —
did not hold here. Every deliberation protocol tested performed worse than simply asking
one capable judge to pick the strongest draft.
A strong selection baseline has to come first
The paper’s framing matters as much as its numbers: much prior work compares
deliberation against a single model’s raw output. Measured instead against best-of-N
selection, the advantage disappears. If you are evaluating an orchestration layer,
the baseline you choose largely determines the answer you get.
Cost belongs in the evaluation
Deliberation ran at 1.5–2.5x the computational cost. The authors report the baseline
holding roughly a 15x better cost-quality ratio, and offer a selection framework:
best-single selection where cost matters or quality matters most, and a single strongest
model where latency dominates.
Read the scope carefully
This is one controlled benchmark, with specific judge models and specific protocol
implementations. It is a strong negative result within that setting — not a general proof
that multi-agent architectures fail. The authors themselves flag judge-dependence as a
limitation. Treat it as a reason to test your own baseline, not as a reason to abandon
a working system.
Abstract
Multi-agent systems in which large language models deliberate toward consensus have
attracted considerable attention, but their practical advantage over simpler methods has
not been closely examined. DeliberationBench is a controlled benchmark that evaluates
three deliberation protocols against a strong baseline: selecting the best response from
a pool of model outputs. Across 270 questions and three independent seeds — 810
evaluations in total — the authors report a striking negative result. The best-single
baseline achieves an 82.5% ± 3.3% win rate, well ahead of the strongest deliberation
protocol at 13.8% ± 2.6%. The 6.0x gap is statistically significant (p < 0.01) and the
deliberation protocols cost 1.5–2.5x more to run. The findings challenge the assumption
that added complexity improves quality in multi-LLM systems.
Paraphrased for readability.
Read the
original abstract on arXiv.
Research metadata
Cite this work
@misc{kaushal2025deliberationbench,
title = {DeliberationBench: When Do More Voices Hurt? A Controlled
Study of Multi-LLM Deliberation Protocols},
author = {Kaushal, Vaarunay and Singh, Taranveer},
year = {2025},
eprint = {2601.08835},
archivePrefix = {arXiv},
primaryClass = {cs.CL},
doi = {10.48550/arXiv.2601.08835},
url = {https://arxiv.org/abs/2601.08835}
}Related field notes
- RLHF, explained: how human feedback shapes model behaviour
- Annotation quality at scale: building reliable QA
- Why AI projects stall — and how to fix your data pipeline
This field note summarises independent research published on arXiv.
BergLabs’ connection to the work is the first author’s affiliation. Nothing here describes
a BergLabs product, benchmark or customer result.
