nestonexStart a project What actually stops an AI feature doing something it should not?
Constraints the system enforces, not instructions the prompt requests. A guardrail is code that checks input before it reaches the model, validates output before it reaches anyone, and limits what any action can do. A sentence in a prompt asking the model to behave is a preference, not a control.
A prompt saying 'never give financial advice' is a request. It works most of the time, which is exactly what makes it dangerous: it produces enough compliance to look like a control while failing on the inputs that were trying to make it fail.
A guardrail is enforced outside the model. A classifier on the output that blocks a response before it is shown. A schema that rejects a malformed action. A policy check that refuses a transfer above a limit regardless of what anything proposed. A spend cap that stops the feature entirely.
The test is simple and worth applying to every safety claim in a design: if the model were actively trying to violate this, would it still hold? If the answer is no, it is a request.
Input. Check what reaches the model: length, rate, whether this user is permitted to ask, and whether the content is something the feature should engage with at all. Cheap, and it catches a surprising share of problems before any tokens are spent.
Structure. Constrain the shape of the output. A response required to match a schema — an enum, a bounded number, a set of allowed fields — cannot express most of the things you were worried about. Structure is the most underused guardrail available.
Output. Check what comes back before anyone sees it: a classifier for prohibited categories, a citation check where sources were required, a refusal check where refusal was expected. This is also where you catch the model asserting something the retrieved passages never said.
Action. The one that actually matters when a model can do things. Limits, allow-lists, approvals, and an audit trail. Every constraint here is one that holds regardless of how the model was persuaded.
A spend cap is a guardrail, and it is usually the one missing. A feature with no per-user ceiling can be turned into a bill by a loop, a scraper or an enthusiastic customer, and the first anyone knows is the invoice.
Caps belong per user and per time window, not only in aggregate. A global monthly cap protects the budget by denying service to everyone at once, which is a poor way to find out.
Rate limits do the same job for latency and abuse. They are also the simplest defence against an automated attempt to find an input that breaks the other layers, because probing takes volume.
Where a model reads untrusted text and can then act, assume the text is trying to steer it. Documents, emails, web pages and chat messages are all inputs an attacker can write, and the model cannot reliably tell content from instruction.
Nothing in the prompt fixes this. What fixes it is the action layer: the manipulated model still cannot exceed a limit, still cannot send to an address outside the allow-list, still cannot skip an approval.
Which means the guardrail design question is not 'how do we stop it being manipulated' but 'what can it still do if it is'. That question has a checkable answer, and it is the one worth writing into a proposal.
ProTrader's coach is a case of guardrails defining the product rather than restricting it. It is given each member's own computed broker metrics and nothing else to talk about, refuses trade recommendations outright, and runs under a daily spend cap per member. It reviews what happened; it does not predict what will.
Axon Wallet enforces the action layer strictly. A model turns plain language into a typed intent and does nothing else; the intent passes JSON Schema validation, policy limits and a two-factor gate before any transaction reaches a chain. The model is never the thing that authorises.
As one layer, not as the layer. Provider filters target broad categories of harm and know nothing about your permissions, your limits or your domain. They will not stop a model recommending something your regulator forbids, or acting beyond a threshold you never told them about.
Badly designed ones do — over-broad blocking produces refusals on legitimate requests, which trains users to route around the feature. That is why the refusal cases belong in the evaluation set alongside the answers, so over-blocking shows up as a measurement rather than as churn.
Outside the model call, in your own application, where it can be read, tested and reviewed by someone who has never thought about prompts. Guardrails implemented as prompt text cannot be unit tested and change behaviour every time the prompt is edited.
It is a useful layer and not a sufficient one. A checking model has its own failure modes and can be manipulated by the same input. Use it to catch categories that are hard to express as rules, and keep deterministic limits underneath it for anything that matters.