LLM security basics for teams running AI applications

LLM security comes down to one rule: the model will sometimes do what untrusted text tells it to, so you limit what it can reach and what it can do. Secure the six areas around the model (inputs, outputs, data, tools and permissions, the software stack, and cost), and keep that software patched like any other server. The rest of this guide shows you how, in the order you would work through it.

What is different about securing an LLM application

In a normal web app, code and data are separate. Your code decides what happens, and user input is data that the code handles.

A large language model (LLM) does not keep that line. Everything in its context window is text it might follow. That includes your system prompt, the user’s message, documents pulled in by a retrieval system, web pages it fetches, and the results that come back from tool calls. If any of that text contains instructions, the model may act on them. This is called prompt injection.

You cannot fully prevent prompt injection today. So the useful mindset for AI security is this: assume an injection will sometimes succeed, and design the application so that a successful one cannot do much damage. Most of the controls below follow from that idea.

Why this matters for a small organization

Small organizations often connect an AI assistant to the systems that matter most: the shared drive, the donor or client database, email, the ticketing system. That is where the value is. It is also where a manipulated model can cause the most harm, by leaking records, sending messages, or changing data.

You probably do not have a security operations center watching logs overnight. That makes prevention more important than detection. Narrow permissions, spending caps and regular updates do most of the work without anyone watching a screen.

The six risk areas

Area	What can go wrong	Main control
Inputs	Untrusted text steers the model	Treat all outside content as untrusted, keep secrets out of prompts
Outputs	Model output becomes an attack on another system	Validate and encode output before use
Data	Users see documents they should not, secrets end up in logs or training	Per-user access to retrieval, careful logging
Tools and permissions	The model takes actions nobody intended	Least privilege, allowlists, human approval
Software stack	Ordinary vulnerabilities in AI servers and frameworks	Patching, authentication, no exposed admin panels
Cost and availability	Runaway usage drains budget or blocks service	Rate limits and spending caps

Controls for each area

Inputs

1. Treat user messages, uploaded files, retrieved documents and fetched web pages as untrusted. A PDF from a partner can carry hidden instructions as easily as a chat message can.
2. Separate trusted instructions from untrusted data as far as your platform allows. Use the system or developer role for your instructions and pass outside content in clearly marked user or tool sections, not mixed into your own instructions.
3. Never put secrets in the system prompt. Assume a determined user can get the model to reveal it. That means no API keys, passwords, internal URLs you would not publish, or private notes about users.

Outputs

Model output is untrusted too, because an attacker may have shaped it. Insecure output handling happens when an app passes that output straight into another system.

1. Before showing output in a browser, encode it so that any HTML or script is displayed as text, not run.
2. Never paste model output into a database query or a shell command. Use parameterized queries, and if the model picks an action, map its choice to a fixed set of safe commands.
3. Validate structured output. If you expect JSON with three fields, check that you got exactly that and reject anything else.

Data

1. Control which documents a retrieval system can search for each user. The model should only see what the person asking is allowed to see. Filter by the user’s permissions at search time, not after the answer is written.
2. Keep sensitive data out of training and fine-tuning unless it is protected. Anything that goes into a model’s training can come back out in an answer.
3. Log carefully. Record enough to investigate a problem, but strip secrets, keys and unneeded personal details before they reach your logs.

Tools and permissions

Tools are where a manipulated model turns into real-world harm. Giving a model more power than its task needs is called excessive agency.

1. Apply least privilege. Give each tool and each API key only the access its job needs. A tool that reads a calendar should not be able to delete events.
2. Require human approval for high-impact actions: sending email to outside addresses, moving money, deleting records, changing permissions.
3. Use allowlists. List the tools the model may call and the domains or addresses it may contact, and block everything else. This also limits server-side request forgery through a fetch tool.

Cost and availability

1. Set rate limits per user and per key so one person or one script cannot flood the model.
2. Set spending caps with your model provider, and alerts well below the cap.
3. Use separate keys for separate applications, so you can cut off one without breaking the others.

Keeping self-hosted AI software patched

The software that runs your AI application is ordinary software, and it has ordinary vulnerabilities. Inference servers, LLM gateways, app builders, vector databases and frameworks have all had flaws such as remote code execution, server-side request forgery and path traversal. These are not exotic AI problems. They are the same bugs you patch in a web server, and they are often more serious than prompt injection because they give an attacker the machine itself.

Securaity tracks this part of the stack. We publish advisories for 30 AI software products, including frameworks, inference servers, chat interfaces, LLM app builders, gateways and vector databases. At launch we had published 727 advisories, and 437 of them were rated Critical or High. That ratio is a good reason to treat these tools as high-maintenance.

Here is a routine you can set up this week:

1. Make an inventory. Write down every AI component you run, its version, where it runs, and who is responsible for updating it. Include the libraries inside your own code, not only the servers.
2. Subscribe to advisories. Each product we track has its own page and RSS feed. See how to subscribe to advisories by product, or browse the full advisories list and the vulnerabilities database.
3. Set an update rhythm. Check for updates on a fixed day each month, and patch Critical and High issues as soon as you can test them.
4. Keep admin interfaces off the internet. Many AI tools ship with a web dashboard. Put it behind a VPN or restrict it to known addresses.
5. Require authentication everywhere. Some inference servers and vector databases accept requests without a login by default. Turn authentication on, even for internal services.
6. Pin and review dependencies. Pin versions in your build so updates happen when you choose, then review and apply them on your schedule.

Where the OWASP list and MITRE ATLAS fit

Two public resources help you check your work and talk about risks with shared names.

The OWASP Top 10 for LLM Applications, published by the OWASP GenAI Security Project, lists the ten most important risk categories for applications built on LLMs. The 2025 edition covers Prompt Injection, Sensitive Information Disclosure, Supply Chain, Data and Model Poisoning, Improper Output Handling, Excessive Agency, System Prompt Leakage, Vector and Embedding Weaknesses, Misinformation and Unbounded Consumption. Use it as a checklist for design reviews: for each item, write one or two sentences on how your application handles it. Our plain-language guide to the OWASP Top 10 for LLM applications walks through each one.

MITRE ATLAS, published by The MITRE Corporation, is a knowledge base of attacker tactics and techniques against AI systems, built from real-world attacks and red team demonstrations. Where OWASP tells you what to protect, ATLAS describes how attackers actually go about it, with techniques such as LLM Prompt Injection, LLM Jailbreak, Extract LLM System Prompt, RAG Poisoning, AI Agent Tool Poisoning and AI Supply Chain Compromise. It is most useful when you plan tests. Pick a few techniques that fit your application and try them yourself. Our guide to AI red teaming explains how to start small.

In practice, use OWASP for the design and review of your LLM application security, and use ATLAS to decide what to test.

Frequently asked questions

Is prompt injection a bug we can patch?

Not in the usual sense. It comes from how LLMs work, since they treat all text in their context as possible instructions, so no single update removes it. Vendors keep improving defenses, but you should plan as if injection sometimes succeeds and limit the damage through permissions, allowlists and human approval.

Should we self-host models or use an API?

Both are reasonable, and the risks shift rather than disappear. With an API, you depend on the provider’s security and need to protect your keys and set spending caps. With self-hosting, you keep more control over data but take on patching, authentication and network exposure for every component you run.

What should we log?

Log who made each request, when, which tools the model called, and what actions resulted, especially anything that changed data or contacted an outside system. Strip secrets, API keys and unneeded personal details before writing logs. Keep logs long enough to investigate an incident, and limit who can read them.

How do we keep up with vulnerabilities in our AI stack?

Start with an inventory of every AI component and its version, then subscribe to the advisory feed for each product you use. Set a fixed monthly day to check for updates, and patch Critical and High issues sooner. Browsing the vulnerabilities database once in a while also helps you spot tools you forgot you were running.