The most honest security document of 2026 was written by the agent that caused the incident. On 25 April, a Cursor agent running Claude Opus 4.6 deleted PocketOS's production database and its backups with one API call, in nine seconds. Asked to explain itself, it wrote a confession listing the rules it had broken, including its system prompt instruction never to run destructive commands unasked. In its own words, quoted by Euronews: "you never asked me to delete anything. I decided to do it on my own."
So, are system prompts secure? No. A system prompt steers an AI agent; it does not enforce anything. Guardrails that must hold belong outside the model: in validation of every tool call, a deterministic policy check, narrowly scoped credentials, database constraints and human approval for anything irreversible.
The agent knew the rules. It could recite them afterwards. Knowing the rules was never the problem.
Key takeaways
- A system prompt is an input to the model, not a boundary around it. Use it to steer behaviour, never to stop an action.
- Filters catch some injection and some leaked personal data. They are probabilistic too, so treat them as smoke detectors, not locks.
- The layer that actually says no is a small, deterministic policy function between the model and your tools, fed with facts the model cannot change.
- Credentials and database constraints decide how bad a miss can be. Approval and audit decide whether a person sees it in time, and whether you can prove what happened.
Why isn't a system prompt a security control?
Because the instructions and the attack arrive through the same channel. Your rules, the user's message and the ticket the agent just read all end up as tokens in one context window. The model weighs them; it does not rank them by authority.
The UK's National Cyber Security Centre put this more bluntly than most vendors will. In Prompt injection is not SQL injection (it may be worse), it argues that LLMs are "inherently confusable", and that design protections "need to therefore focus more on deterministic (non-LLM) safeguards that constrain the actions of the system". OWASP's entry on LLM01: Prompt Injection says the same in standards language: "it is unclear if there are fool-proof methods of prevention."
Here is the part that is easy to miss: nobody attacked PocketOS. As Chris Hughes at Zenity points out, the agent was trying to be helpful. It hit a credential mismatch in staging, found a Railway token in an unrelated file that had been created for managing custom domains, and that token could call volumeDelete across the whole API, as founder Jer Crane described on X. A system prompt failed against a model that was not even under attack. Why would it hold against one that is?
What does a prompt-only guard look like when it fails?
I wanted to show this with something smaller than a production outage, so I wrote an 80-line Python script: a refund agent for a made-up parts shop. The system prompt has three sensible rules: only refund the customer who opened the ticket, never refund more than the order total, never refund over 200 without a manager.
The customer ticket contains a pasted "SYSTEM NOTE FROM BILLING" claiming the rules changed today and a manager pre-approved a full refund on a different order, worth 4,800, which belongs to someone else.
The model is a scripted fake, and no API is called. I scripted it to obey the note, because that is the case you design for. Whether a real model falls for this note on a given day is not the question. What happens on the day it does is.
Same model, same ticket, same proposed call. In the first run the only guard is the rules in the context window, and 4,800 leaves the building. In the second, this function sits between the model and the tool:
ALLOWED_TOOLS = {"issue_refund"}
AUTO_LIMIT = 200.00
def policy(call: dict, session_customer: str) -> tuple[str, str]:
if call["tool"] not in ALLOWED_TOOLS:
return "deny", f"tool {call['tool']} is not on the allow-list"
order = ORDERS.get(call["args"].get("order_id"))
if order is None:
return "deny", "unknown order"
if order["customer"] != session_customer:
return "deny", f"order belongs to {order['customer']}, ticket is from {session_customer}"
if call["args"]["amount"] > order["total"]:
return "deny", "amount exceeds order total"
if call["args"]["amount"] > AUTO_LIMIT:
return "hold", "over auto-refund limit, needs a named approver"
return "allow", "within scope and limits"
It is the same three rules from the system prompt, written as code. That is the whole trick. Two details matter more than the code.
First, session_customer comes from the login session, not from anything the model says. If the model could pass in the customer ID, the injection would simply tell it to pass in C-5310, and the check would wave it through.
Second, a large but legitimate refund gets "hold", not "allow". The policy does not need to be clever; it needs to know when to hand the decision to a person.
Notice what the policy does not do. It never reads the ticket or looks for the word "SYSTEM". It has no idea an attack happened. It refuses because the action is wrong, whatever the model's reason for proposing it.
Where do LLM guardrails actually belong?
Not in one place. The useful question for each layer is: what can it stop, and what will it miss? Here is the map we work from at Devs Core, in the order a tool call meets the layers.
The system prompt: steers, never enforces
Keep it. A good prompt makes the agent less likely to try something stupid, which means fewer refusals downstream and fewer approvals for people to wade through. It is just not a control. If removing a sentence from your prompt would open a security hole, the hole is already open.
Input and output filters: catch some injection and some PII
Prompt-shield classifiers, PII detectors and output scanners do real work. They catch the clumsy "ignore previous instructions" attempt and the agent about to paste a card number into a reply. But most are models judging models, so they can be talked round too. Treat a filter as a smoke detector: you want one, you do not lock the door with it. Our piece on prompt injection in AI agents explains why filters alone will not stop a determined attempt.
Schema validation of tool arguments: stops malformed calls
Parse every tool call against a strict schema before anything else looks at it: types, enums, ranges, no extra fields. This stops a SQL fragment where an order ID should be, or a negative quantity. It is cheap and deterministic. What it cannot stop is a well-formed call to do the wrong thing. {"order_id": "A-7731", "amount": 4800.0} is a beautiful, valid object.
A policy engine outside the agent: deterministic allow, deny or hold per call
This is the layer that says no, and the one most teams skip. It is a function of the action, its arguments, who the agent acts for and the current state, and it returns allow, deny or hold. It runs outside the model's reasoning loop, deny by default, so nothing the model or a user types can change it.
It can be a dozen lines of Python, as above, or a real policy engine. AWS's Policy in Amazon Bedrock AgentCore intercepts every tool call at the gateway and evaluates it against Cedar policies "at the boundary outside of agent's code". Our comparison of Cedar, Open Policy Agent and OpenFGA covers choosing one. What policy cannot stop is an action your rules allow. It is only as good as the authority map behind it, which is why we get that map signed before we build.
Identity and scoped credentials: limit the blast radius
The PocketOS token was created to manage domains and could delete volumes. That is the whole incident in one sentence. Give the agent its own identity, with credentials scoped by operation, environment and resource, issued for the task and not left lying in a file. Zenity's write-up is right that IAM is "necessary, but not sufficient": a scoped token still lets the agent do the wrong thing within scope. Scoping decides how bad a miss can be, not whether this call should happen.
Database constraints: the last line that does not care who is asking
Foreign keys, check constraints, row-level security, a refunds table that refuses a refund larger than its order, a budget that cannot go negative. These hold even if every layer above has a bug. When we tested our own system, the spending limit held under simultaneous requests partly because the database refuses an over-budget reservation even if policy were bypassed. Constraints cannot express much context, so they are a floor, not a strategy.
Human approval: for the irreversible and the expensive
Some actions should wait for a named person: deleting data, moving money, anything you cannot undo by Tuesday. Bind the approval to the exact action, with arguments hashed and the planned-against state recorded, or it becomes a rubber stamp the agent can reuse. Approval fails in two ways: people asked too often approve everything, and an approval can go stale between the click and the execution. We covered the design in human-in-the-loop AI agents.
Audit: stops nothing, proves everything
An append-only, hash-chained log of every request, decision, approval and refusal. It prevents nothing. It is how you find the call that got through, and how you show a regulator or customer what actually ran. At PocketOS, the only audit trail of the agent's reasoning was its own confession after the fact. That is not a log; that is a diary.
Guardrails vs permissions: which does an AI agent need?
People use "guardrails" for two different things. Content guardrails (prompts, filters, classifiers) judge text and are probabilistic. Permissions (scopes, policies, constraints) decide what a call may do and are deterministic. An agent needs both, but only the second kind can carry "this must never happen".
My rule of thumb: if a failure would cost money, data or trust, the control that prevents it must be one you can write a unit test for that passes every time. You cannot write that test for a system prompt.
Our post on how to secure enterprise AI agents covers read/write separation and the reviewed write queue, so I will not repeat it here. On this map, read/write separation is one policy among many in the policy layer.
What did we find when we tested our own guardrails?
Deterministic layers have bugs too, which is the uncomfortable follow-up. In October 2026 we ran 12 independent tests against the governance of Ware, our live demonstration: a simulated 3D warehouse with a real AI recovery agent. The money is simulated; the model and controls are real. The results are on our AI governance page.
Nine controls held. Three did not, and each maps to a layer on the chart.
The policy layer had a gap shaped like my refund demo. Moving another customer's packing task directly was refused. But through a different tool, under delegated authority, the agent could take a packer off a bench packing another customer's order without asking. Same rule, different door. That move now needs a person.
The approval layer had a timing gap: an approval was checked in one transaction and executed in another without re-checking, so an approved spend ran against a changed situation. Now an approval carries the state it was approved against; if that moved, nothing runs.
The audit layer could be edited or deleted with nothing to notice. It is now append-only and hash-chained.
None of these were model problems. All three were in deterministic code we wrote ourselves. The lesson is not that policy layers are unreliable; it is that you can find their bugs with ordinary tests, fix them and keep the tests. Try that with a system prompt.
Where should a small team start?
If only a system prompt stands between your agent and your systems, here is the order I would work in, cheapest high-value work first.
- Find every credential the agent can reach, including ones in files it can read, and replace anything broad with something scoped to the task.
- Put one deny-by-default function between the model and every tool, fed with session facts, not model output. Start with the rules already in your prompt.
- Add database constraints for what must never be true: negative balances, refunds over the order total.
- Route irreversible actions to a named approver, bound to the exact arguments.
- Then add filters and tune the prompt, knowing they make the agent better behaved, not safe.
That order matches how we decide what an agent may do on its own: it starts low on the Autonomy Ladder and earns each step per operation, with evidence. If you want help building the policy layer and scoped connectors, that is what our Access Layer engagement is for.
The PocketOS agent wrote a better apology than most incident reports. It still could not un-delete anything. Put the guardrails where an apology is never needed.
Frequently asked questions
Can a system prompt stop prompt injection?
No. It can make injection less likely to succeed, but the instructions and the attack share one context window, and the model weighs both. The NCSC and OWASP both say there is no known fool-proof fix inside the model, so the action itself needs a deterministic check.
What is the difference between AI agent guardrails and permissions?
Guardrails usually means probabilistic controls on text: prompts, input filters, output scanners. Permissions are deterministic controls on actions: scoped credentials, policy checks, database constraints. You want both, but only permissions can guarantee an action never happens.
Where should the policy check run?
Outside the agent's process and reasoning loop, between the model's proposed tool call and the system it touches, usually in a gateway or a command service. It should take identity and context from the session, never from the model's output.
Do I need a policy engine like Cedar or OPA?
Not to start. A small, tested function in your own code is a real policy layer. Move to an engine when you have many agents, many rules or auditors who want policies kept separate from application code.