Content moderation needs a versioned policy, preserved context and an appealable decision. Models can triage risk and route languages, while trained reviewers handle contextual cases and specialists handle severe risks. Evaluation should measure missed harm as well as incorrect enforcement.
Composite reference architecture. A modelled trust-and-safety workflow; no removal rate or safety outcome is asserted.
The operating problem
A classifier score cannot resolve context, policy exceptions, local language, appeals or changes in what the platform permits.
How the workflow works
Attach the content, report, relevant account context and policy version to one review case. Duplicate matching and model signals can prioritise the queue, but the reviewer needs the exact rule and contextual evidence behind a proposed action. Record whether content was allowed, limited, removed or escalated, with the reason shown through the appropriate user process.
Where review and controls matter
Language and cultural context can change a decision. Route uncertain cases to qualified reviewers and provide the required wellbeing support for harmful material. Appeals need access to the original evidence and an independent review path where the policy calls for one. Use corrected decisions to review policy wording and model errors; appeal overturn alone cannot measure all mistakes because many users never appeal.
Risk reference: NIST AI RMF.
What to measure
Use a fixed cohort and measurement window. Count numerator events only within the denominator's eligible cohort; sampled measures use the same audit sample for both terms. Record the source, exclusions and owner, and keep missing data visible. These definitions are proposed measurement contracts, not observed results.
| Measure | Definition and unit |
|---|---|
| Policy precision | Correct enforcement decisions divided by audited enforcement decisions; percent, by policy category. |
| Policy recall | Violating items correctly identified divided by violating items in a representative labelled audit set; percent. |
| Severe-content latency | Time from severe-content detection or report to the required protective action; median and p90 minutes. |
| Appeal overturn | Appealed decisions changed after review divided by decided appeals; percent, with appeal participation and backlog. |
| Reviewer agreement | Agreement using a declared task-appropriate statistic on independently reviewed cases; report scale and sample. |
| Harm exposure | Views or impressions of confirmed violating content within the audited cohort divided by all eligible views or impressions in that same cohort; rate per declared unit. State sampling weights for any population estimate. |
A useful first pilot
Evaluate one policy category with a labelled sample containing context-dependent and severe cases. Audit both allowed and enforced content, then test the appeal and specialist escalation paths.
Bring the workflow, its source records and the decision that needs an owner to a BergLabs pilot discussion.
