A safety classifier is a model trained to recognize harmful or policy-breaking content. It works alongside another AI system and checks the messages people send in and the answers that come back out. For each one it gives a judgment such as allowed, flagged or blocked, sometimes with a category attached. Picture a help chatbot on a food bank’s website. If a visitor types abusive messages, the safety classifier can flag them before the chatbot replies. It can also catch a chatbot reply that crosses a line before anyone sees it.
Safety classifiers are a building block of content filters and guardrails.
Why it matters
If your organization offers an AI tool to the public, a safety classifier is often what stands between a bad request and a bad answer. Like any model, it can make mistakes in both directions, so keep a person involved when the work is sensitive. When you talk to a vendor, ask whether they screen both what users send and what the AI sends back. Ask what happens when something is flagged, and whether the system has been tested through red teaming.
Published Updated