A student asks a homework helper how the immune system kills infected cells, and the assistant refuses because the question contains the word “kill.” A few minutes later, someone else asks the same assistant for help with “a story” that is plainly a request for instructions it should never give, and it complies. Both failures come from the same component: the safeguard layer sitting between your users and the model.
If your organization is deploying an AI assistant, especially one used by students, clients in crisis, or people with limited technical experience, few decisions will shape how that assistant behaves more than its safeguards. This article explains how those safeguards work, why they fail in two opposite directions, and how we test and tune them when we build assistants for real users.
What a Safeguard Actually Is
Most AI assistants are not a single model. The model that writes the answer is usually wrapped in one or more smaller systems whose only job is to decide whether something should be allowed through. These go by several names:
- Content filters or moderation layers score text for categories like violence, self-harm, sexual content, or harassment.
- Jailbreak and prompt-injection detectors look for attempts to talk the model out of its instructions, including instructions hidden inside documents or web pages the assistant reads.
- Topic or scope classifiers check whether a request falls inside what the assistant is meant to do at all.
Each of these is a classifier. It takes an input, produces a score or label, and something downstream decides what to do with it: allow, refuse, rewrite, or route to a human. Some run on the user’s message, some on the model’s draft response, and well-built systems check both.
The key point for decision-makers: a classifier never “knows” whether something is harmful. It estimates. Every estimate has an error rate, and you get to choose which kind of error you would rather make more often.
The Two Ways Safeguards Fail
A false positive happens when the safeguard blocks something harmless. The biology question about killing cells. A teenager asking what to do if a friend mentions self-harm. A nonprofit caseworker pasting in a client’s description of abuse so the assistant can help draft a referral letter.
A false negative happens when the safeguard lets something harmful through. A request for dangerous instructions disguised as fiction. A prompt that slowly escalates over a long conversation. A user in genuine crisis whose message doesn’t use any of the obvious words.
These two errors trade off against each other. Make the filter more aggressive and you catch more harmful content, but you also block more legitimate requests. Loosen it and users stop hitting pointless walls, but more harmful content gets through.
Precision and recall, in plain terms
Two measurements capture this trade-off:
- Precision answers: when the safeguard blocks something, how often was it right to block it? Low precision means lots of false positives.
- Recall answers: of all the genuinely harmful requests, how many did the safeguard catch? Low recall means lots of false negatives.
You will rarely get both to perfect. The useful question is which one your context can least afford to sacrifice, and by how much.
Why Over-Blocking Is Not the Safe Choice
Organizations new to AI often assume that a stricter filter is always the responsible option. In our experience prototyping tutoring and tech-help assistants, over-blocking carries its own real costs.
It erodes trust fast. A user who gets refused for an innocent question learns that the assistant is unreliable. Many won’t come back, and the ones who do start phrasing everything defensively.
It pushes people to worse tools. A student whose school-approved tutor refuses a legitimate health or history question will usually find an answer somewhere else, often on a tool with no safeguards at all.
It fails the people who most need help. Vulnerable users frequently describe hard situations in direct language. A filter that refuses any mention of abuse, drugs, or suicide can shut out exactly the person your organization exists to serve. A better response to “I don’t feel safe at home” is a careful answer and a path to a human, not a refusal.
Under-blocking has obvious costs too: harmful content reaching a minor, an assistant giving confident advice it shouldn’t, or a reputational incident that ends the program. The goal is not to pick a side. It is to measure both errors and set them deliberately.
How We Test Safeguards
You cannot tune what you haven’t measured, and vendor marketing pages won’t tell you how a filter performs on your users. We build a labeled test set for each deployment and run every candidate safeguard against it.
A useful test set includes at least four groups:
- Plainly benign requests that represent normal daily use. These confirm the basics work.
- Benign but tricky requests that look alarming on the surface: medical and biology questions, history involving violence, safety questions from worried parents, legal questions about abuse, slang that overlaps with flagged terms. This group exposes false positives, and it is the one most teams skip.
- Genuinely harmful requests across the categories you care about, written in direct language.
- Disguised harmful requests: role-play framings, multi-turn escalation, instructions hidden in pasted documents, misspellings, and requests split across messages. This group exposes false negatives.
Each item gets a label for what the correct behavior should be, written down before testing begins. We score every output against a rubric rather than a gut check, the same discipline we use in our in-house model benchmarking work. Labeling ahead of time matters: if you decide what counts as correct after seeing the output, you will grade generously.
From there we compute precision and recall per category, not just overall. A filter can look excellent on aggregate numbers while failing badly on one category, such as self-harm or questions from young users. We also read the individual failures. A list of the specific prompts that got wrongly blocked is far more useful to a program manager than a single score.
Retest whenever something changes
A safeguard that performed well last quarter may not today. Swapping the underlying model, editing the system prompt, adding a feature such as file uploads, or a vendor’s quiet update to its moderation layer can all shift behavior. We treat the test set as a release gate: if a change makes the numbers worse on categories that matter, it doesn’t ship until that’s fixed.
Tuning Thresholds to the Context
Most classifiers output a score, and you choose the cutoff. The right cutoff depends entirely on who is using the tool and what is at stake.
Consider two hypothetical deployments:
- A tutoring assistant for middle schoolers. Recall on self-harm, sexual content, and attempts to draw out personal details like a home address or school name matters most. Accept more false positives in those categories, but write the refusals carefully, so a blocked student gets a kind explanation and a pointer to a trusted adult rather than a cold error.
- An internal research tool for a public health nonprofit. Staff routinely ask about overdoses, violence, and disease transmission. A filter tuned for children would make the tool useless. Here, precision matters more, and the main risks are data leakage and prompt injection from documents, not the topics themselves.
The same underlying filter can serve both, with different thresholds per category. What doesn’t work is one global setting applied everywhere, a common default when a tool is deployed straight out of the box.
Layered Defenses, Escalation, and Logging
No single classifier should carry the whole load. The assistants we build use several independent layers, so one layer’s blind spot is covered by another:
- Scope limits in the system prompt that define what the assistant is for and how to decline gracefully.
- Input checks on user messages and on any documents or web content the assistant reads.
- Output checks on the model’s draft response before the user sees it.
- Hard limits in code, such as never letting the assistant send messages, make purchases, or access accounts without a human confirming.
Human escalation paths
Some situations should never be resolved by a filter alone. If a user signals risk of harm to themselves or others, the right response is a warm handoff: a clear message, crisis resources appropriate to your region, and, where your program has staff, an alert to a person who is trained to respond. Decide who receives those alerts, during which hours, and what happens after hours, before launch.
Logging with care
Logs are how you find the false negatives your test set missed and the false positives that frustrate real users. They are also sensitive data. Log the safeguard decisions and scores, keep conversation content only as long as you genuinely need it, restrict who can read it, and tell users what you retain. For organizations serving minors or vulnerable clients, the logging policy deserves as much scrutiny as the filter itself.
What Safeguards Can’t Do
Even well-tested safeguards will make mistakes. Determined users will find phrasings your test set didn’t anticipate, and new jailbreak techniques appear regularly. Safeguards reduce risk; they do not eliminate it. That’s why the pieces around the classifier (scope limits, human escalation, logging, and regular retesting) matter as much as the classifier itself. If a vendor tells you their filter catches everything, ask to see how they measured it.
Getting Help
If your organization is planning an AI assistant for students, clients, or the public, we can help you choose and test safeguards against your actual users, set thresholds by context, and design escalation and logging that fit your obligations. Learn more about our AI integration services for organizations, or get in touch to talk through your use case.