Marketplace moderation should combine content, seller context and policy evidence to prioritise harmful activity while preserving legitimate supply. Unusual inventory needs contextual review, and consequential enforcement needs a recorded reason and an appeal route.
Composite reference architecture. A reference marketplace-safety design; no client deployment or enforcement outcome is claimed.
The operating problem
Marketplace content mixes products, reviews, seller messages and media. Rare legitimate items resemble abuse, while coordinated bad actors learn simple rules.
How the workflow works
Join the listing or review with its media, reports and relevant seller history. Multimodal signals can detect suspicious patterns, and coordinated-account evidence can change queue priority. Keep the policy basis and source timestamps visible. A product's rarity or an unfamiliar visual style should not by itself establish a violation.
Where review and controls matter
Specialists review severe harm, coordinated abuse and difficult legal context. Seller-level actions require evidence that supports the seller-level conclusion, rather than inheriting a single uncertain item score. Preserve legitimate counterevidence and appeal outcomes. Repeated abuse feeds back into detection rules, while sampled allowed content helps identify harm that the queue never caught.
Risk reference: NIST AI RMF.
What to measure
Use a fixed cohort and measurement window. Count numerator events only within the denominator's eligible cohort; sampled measures use the same audit sample for both terms. Record the source, exclusions and owner, and keep missing data visible. These definitions are proposed measurement contracts, not observed results.
| Measure | Definition and unit |
|---|---|
| Harm exposure | Impressions of confirmed violating marketplace content within the audited cohort divided by all eligible impressions in that same cohort; rate per declared unit. State sampling weights for any population estimate. |
| Enforcement precision | Correct enforcement decisions divided by audited enforcement decisions; percent. |
| Detection recall | Violating cases correctly identified divided by violating cases in a representative labelled audit set; percent. |
| Time to severe review | Time from report or detection to qualified review for severe cases; median and p90 minutes. |
| Appeal overturn | Appealed decisions changed after review divided by decided appeals; percent, with appeal coverage. |
| Repeat abuse | Previously enforced actors with a new confirmed violation during a fixed follow-up window divided by eligible previously enforced actors whose full window has elapsed; percent. |
A useful first pilot
Select a defined marketplace policy area and sample enforced and allowed content. Include unusual legitimate products and coordinated abuse. Compare error types, severe-review delay and appeal outcomes before widening the scope.
Bring the workflow, its source records and the decision that needs an owner to a BergLabs pilot discussion.
