TLDR
DeliberationBench compared three multi-LLM deliberation protocols against a best-single selection baseline across 270 questions and three seeds. Best-single selection won 82.5% of comparisons. The strongest deliberation protocol, senate-style debate, won 13.8% at 1.5 to 2.5 times the compute cost. Adding structured deliberation made results worse, not better.
Research / Multi-LLM systems
A controlled study finds that selecting the best single response can outperform more complex multi-LLM deliberation protocols.
DeliberationBench compared three multi-LLM deliberation protocols against a strong best-single selection baseline across 270 questions and three seeds. The baseline won 82.5% of comparisons; the strongest deliberation protocol won 13.8% – a 6.0x gap at 1.5–2.5x the computational cost.
The key finding
Across every protocol tested, adding structured deliberation between models made results worse, not better – and cost more to run.
Win rates, mean ± standard deviation over three seeds, N = 270. All differences statistically significant at p < 0.01.
What was tested
Every approach starts from the same five candidate drafts, generated by the same model council. What differs is how a winner gets chosen. That design isolates the value of deliberation itself, rather than the value of having several candidates to choose from.
| Approach | How a winner is chosen | Win rate |
|---|---|---|
| Best-single selection baseline | A judge model compares all five drafts in a single prompt and picks the best. | 82.5% ± 3.3% |
| Senate debate v2-B | Each draft is assigned a defender agent. They hold a structured debate – opening statements, rebuttals, closing arguments – then a final judge selects the winner. | 13.8% ± 2.6% |
| Blind ranking v1 | Drafts are shuffled to strip model identity, then a deliberation agent ranks all five and returns the top-ranked response. | 3.2% ± 1.8% |
| Rubric scoring v2-A | A deliberation agent scores each draft against multiple criteria; the highest aggregate score wins. | 0.6% ± 1.0% |
The benchmark itself is 270 questions spanning factual knowledge, reasoning and domain-specific expertise, each with a manually researched reference answer, and deliberately weighted toward medium and hard items.
What the result means
More agents do not automatically produce a better answer
The intuition behind multi-agent systems – that committees outperform individuals – did not hold here. Every deliberation protocol tested performed worse than simply asking one capable judge to pick the strongest draft.
A strong selection baseline has to come first
The paper’s framing matters as much as its numbers: much prior work compares deliberation against a single model’s raw output. Measured instead against best-of-N selection, the advantage disappears. If you are evaluating an orchestration layer, the baseline you choose largely determines the answer you get.
Cost belongs in the evaluation
Deliberation ran at 1.5–2.5x the computational cost. The authors report the baseline holding roughly a 15x better cost-quality ratio, and offer a selection framework: best-single selection where cost matters or quality matters most, and a single strongest model where latency dominates.
Read the scope carefully
This is one controlled benchmark, with specific judge models and specific protocol implementations. It is a strong negative result within that setting – not a general proof that multi-agent architectures fail. The authors themselves flag judge-dependence as a limitation. Treat it as a reason to test your own baseline, not as a reason to abandon a working system.
Abstract
Multi-agent systems in which large language models deliberate toward consensus have attracted considerable attention, but their practical advantage over simpler methods has not been closely examined. DeliberationBench is a controlled benchmark that evaluates three deliberation protocols against a strong baseline: selecting the best response from a pool of model outputs. Across 270 questions and three independent seeds – 810 evaluations in total – the authors report a striking negative result. The best-single baseline achieves an 82.5% ± 3.3% win rate, well ahead of the strongest deliberation protocol at 13.8% ± 2.6%. The 6.0x gap is statistically significant (p < 0.01) and the deliberation protocols cost 1.5–2.5x more to run. The findings challenge the assumption that added complexity improves quality in multi-LLM systems.
Paraphrased for readability. Read the original abstract on arXiv.
Research metadata
Cite this work
@misc{kaushal2025deliberationbench, title = {DeliberationBench: When Do More Voices Hurt? A Controlled Study of Multi-LLM Deliberation Protocols}, author = {Kaushal, Vaarunay and Singh, Taranveer}, year = {2025}, eprint = {2601.08835}, archivePrefix = {arXiv}, primaryClass = {cs.CL}, doi = {10.48550/arXiv.2601.08835}, url = {https://arxiv.org/abs/2601.08835} }Related field notes
- RLHF, explained: how human feedback shapes model behaviour
- Annotation quality at scale: building reliable QA
- Why AI projects stall – and how to fix your data pipeline
This field note summarises independent research published on arXiv. BergLabs’ connection to the work is the first author’s affiliation. Nothing here describes a BergLabs product, benchmark or customer result.
