Ecommerce search-quality work should connect buyer intent, query-product judgements and catalogue corrections to a stable evaluation set. BergLabs reports programme-level improvements for an anonymised e-retail engagement. Those figures are first-party reports; the workflow below is an explanatory design, not an independently audited causal account.
Anonymised documented outcome. BergLabs' public case-study page attributes these figures to delivery decks. They are source-reported, not independently audited current performance. The underlying denominator, sample design and complete measurement method are not publicly disclosed.
What the reported programme evidence shows
BergLabs' public case-study page reports search pain moving from 46% to 26%, more than 2.5 million query-product pairs labelled, more than 1.2 million catalogue corrections and over $15 million in stated annual revenue lift. The 46% to 26% change is 20 percentage points. The original search-pain definition, full measurement window and attribution method are not supplied here. These are source-reported programme figures, not a new measurement or proof of incremental causation.
The operating problem
Search pain persists when teams optimise ranking without repairing query intent, catalogue attributes and the feedback labels that train evaluation.
How the workflow works
Sample queries and result sets by intent, frequency and known failure type. Reviewers judge the query-product relationship using an agreed relevance scale and record whether the defect belongs to ranking, catalogue attributes or missing product evidence. Accepted judgements become evaluation data, while catalogue errors route to the team able to correct them.
Where review and controls matter
Clicks and purchases carry position and exposure bias, so they should not be treated as unquestioned relevance labels. Preserve a stable evaluation sample and report results by query segment. Review policy-sensitive advertisements separately from organic relevance decisions. Revenue comparisons also need an attribution method and controls for seasonality, promotions and catalogue changes.
Risk reference: NIST AI RMF.
What to measure
Use a fixed cohort and measurement window. Count numerator events only within the denominator's eligible cohort; sampled measures use the same audit sample for both terms. Record the source, exclusions and owner, and keep missing data visible. These definitions are proposed measurement contracts, not observed results.
| Measure | Definition and unit |
|---|---|
| Search pain | Source-specific measure requiring the original rubric and denominator. No formula is inferred from the reported percentage. |
| Judged relevance or nDCG | Mean query-level nDCG at a declared cutoff k, relevance scale and treatment of unjudged results; score from 0 to 1. |
| Zero-result rate | Searches returning no eligible products divided by eligible searches; percent. |
| Click-through rate | Searches with a qualifying result click divided by eligible searches; percent, with position and surface rules. |
| Conversion | Search sessions producing a qualifying purchase divided by eligible search sessions; percent, within a declared window. |
| Revenue attribution | Recognised revenue linked to the evaluated cohort under the stated attribution method; currency. Incremental lift requires a suitable comparison design. |
A useful first pilot
Begin with a fixed query cohort and a documented relevance rubric. Measure ranking quality and catalogue defect resolution separately, then join them to a defined conversion cohort without assuming every revenue change came from annotation.
Bring the workflow, its source records and the decision that needs an owner to a BergLabs pilot discussion.
