Cost
OpenAI's moderation guide says the moderation endpoint is free to use. That is vendor-reported. A separate billed classifier would be an extra request. This card does not add a price for one.
Map · Ship & Maintain · 07 · Safety
Concept card · Ship & Maintain
Safety
Safety is the limit on harmful or disallowed use, enforced in your product and by the provider's model and policies.
Limits on harmful or disallowed use, in the product and in the model.
What changed
Oct 18: First card. The picture shows a product check before the model, on a made-up allowed question and a request the policy disallows. Sources are OpenAI's moderation and safety guides and Anthropic's Usage Policy.
1 What it is
Two different checks can stop a reply. Your product can refuse a request before it calls a model. The provider can refuse, or can flag the text, under its own policy. OpenAI's moderation guide says you can classify text and images for potentially harmful content, and that you should treat the scores as signals for your application's policy, not as an automatic block. The moderation guide says the moderation endpoint is free to use. OpenAI's safety practices also say to red-team the product, to keep a person in the loop for high-stakes uses, and to give users a way to report a bad reply. Anthropic's Usage Policy applies to anyone who sends inputs to Claude, including through an API, and says it is meant to prevent products from being used to create or contribute to real-world harm. Anthropic says its safeguards may block or limit some outputs, and that a block by itself is not a finding that the Usage Policy was violated. This card does not describe how to cause harm.
2 Why it matters
Quality
A check that blocks ordinary warehouse questions teaches people to route around the product. Test the check on requests you intend to answer and on requests your policy disallows.
Risk
Shipping with no product check means the only limit is whatever the provider does that day. OpenAI says to use moderation results to enforce your policy. Anthropic's Usage Policy can still lead to a warning or a suspension if use breaks it.
3 How to apply · try today
Write five requests your product should answer and five it should refuse. Run both sets. Record what your product did and what the model did. Adjust the product check where the result disagrees with your policy.
Common mistake
Treating a provider refusal as your whole safety plan, or treating a moderation flag as a block you never review. OpenAI says the flag is a signal. Your policy decides the next step.
Self-check
What is a moderation flag, according to OpenAI's moderation guide?
Support-docs answer bot uses this concept in one build. The set on this tab is still the note below.
Coming soon. Five requests you should answer and five your policy should refuse, with the product's action written next to each.
Coming soon. How OpenAI separates a moderation signal from your block decision, and how Anthropic describes a safeguard block versus a Usage Policy finding.
Questions
- What is safety?
- Safety is the limit on harmful or disallowed use, enforced in your product and by the provider's model and policies.
- Does a moderation flag always block the reply?
- OpenAI's moderation guide says to treat scores as signals for your application's policy, not as an automatic block.
- Is a provider safeguard the same as a policy violation?
- Anthropic's Usage Policy says products include safeguards that may block or limit some outputs, and that a block by itself does not mean the Usage Policy was violated.