Hybrid annotation can reduce routine review when confidence thresholds are validated for each task and risk class. High-confidence items still need independent sampling, and ambiguous or sensitive labels need trained review. BergLabs' reported programme figures describe one source-reported programme, not a universal result of using a 0.9 threshold.
Anonymised documented outcome. BergLabs' public case-study page reports the programme figures from source decks. They have not been independently re-measured here, and the underlying denominator, sample design and complete measurement method are not publicly disclosed.
What is hybrid AI-human annotation?
Hybrid AI-human annotation routes model-assisted labels through human review according to measured risk. Automation handles eligible, well-calibrated cases; specialists resolve ambiguity, sensitive classes and sampled fast-path work before the accepted label becomes authoritative.
What the reported programme evidence shows
BergLabs' public case-study page reports 30% faster turnaround, 15% cost savings and 96% accuracy for the annotation programme. It also describes a confidence threshold above 0.9 for low-risk segments. The threshold is a routing setting, not an accuracy result.
The operating problem
Prelabels can accelerate obvious items, but a single global confidence threshold hides class imbalance, rare-event risk and reviewer disagreement.
How BergLabs operates the route
In our in-house reference pattern, BergFlow keeps each prelabel, confidence score, guideline version, calibration evidence and reviewer decision together. BergAuto applies class-specific routing and audit gates instead of treating one confidence number as a blanket permission to auto-accept.
Quinn supports specialist vetting; Rosterr routes review and escalation work, produces QA outputs and gives operators a dashboard view of responsibilities, review states and delivery pods; Charterr supports managed capacity for governed annotation pods. This is the BergLabs operating pattern; it is not a claim that every client uses the same configuration.
How the workflow works
Attach the model version, guideline version and confidence to each prelabel. Compare confidence with observed correctness on a protected evaluation set, including rare classes. A routing threshold should reflect the cost of a missed error and the evidence available for that class. Specialist corrections must reach the accepted label record and the relevant regression sample.
Where review and controls matter
A score above 0.9 is not a 90% accuracy guarantee. Calibration can change across classes, models and data. Sample fast-path items independently, refresh the gold set without contaminating it and review disagreements for guideline ambiguity. Track accepted units and rework so nominal throughput does not hide correction effort shifted downstream.
Risk reference: NIST AI RMF.
What to measure
Use a fixed cohort and measurement window. Count numerator events only within the denominator's eligible cohort; sampled measures use the same audit sample for both terms. Record the source, exclusions and owner, and keep missing data visible. These definitions are proposed measurement contracts, not observed results.
| Measure | Definition and unit |
|---|---|
| Gold accuracy | Correct labels on protected gold items divided by scored gold items; percent, by class. |
| Accepted yield | Items accepted under final quality checks divided by eligible input items; percent. |
| Review rate | Items receiving human review divided by eligible processed items; percent, split into routing and audit samples. |
| Rework | Items returned for correction divided by reviewed items; percent. |
| Turnaround | Time from eligible item intake to accepted label; median and p90 hours. |
| Cost per accepted unit | All attributable model, review, audit and correction costs divided by accepted units; currency per unit. |
Hybrid AI-human annotation FAQ
Does 0.9 confidence mean 90% accuracy?
No. A confidence score is a model output, while accuracy is an observed result against labelled evidence. The relationship must be calibrated separately by task, class and data distribution.
How should a review threshold be set?
Set thresholds by task and class using a protected evaluation sample, error cost and available reviewer capacity. Re-test them when the model, guideline or source distribution changes.
What should be audited on the fast path?
Sample fast-path items independently and track class-level accuracy, calibration drift, rare-class misses, guideline ambiguity and corrections that reach the accepted record.
A useful first pilot
Choose a bounded task and compare routed review with a fully reviewed reference sample. Lock class-specific thresholds only after measuring errors and review load, and retain a fast-path audit sample throughout the pilot.
Bring the workflow, its source records and the decision that needs an owner to a BergLabs pilot discussion.
