AI security glossary
Short definitions of the terms you will meet in advisories, vendor policies and AI rules, each with a page that explains why it matters.
A
- Acceptable use policy
- A written policy that tells staff and volunteers which tools they may use, for what purposes, and with which data.
- Adversarial example
- An input changed on purpose, often in ways a person would not notice, so that an AI model gets it wrong.
- AI agent
- An AI system that plans and carries out multi-step tasks on its own, using tools like web browsing, email, files or other software.
- AI governance
- The policies, roles and processes an organization uses to decide how AI tools are chosen, used, monitored and reviewed.
- AI incident
- An event where using an AI tool causes harm or nearly does, such as a data leak, a costly mistake or a fraud using fake media.
- AI risk assessment
- A structured look at what could go wrong when your organization uses an AI tool, how likely and serious it is, and what you will do about it.
- AI safety
- The field that works to stop AI systems from causing harm, whether by accident, through misuse, or in ways nobody intended.
- AI security
- The work of protecting AI tools, the data they use and the people who rely on them from attack and misuse.
- AI supply chain attack
- An attack on the parts an AI tool is built from, such as models, datasets, code libraries or plug-ins, rather than on the tool itself.
- Alignment
- The work of making an AI system's goals and behavior match what its developers and users intend, including following instructions and avoiding harm.
- API key
- A secret string of characters that a program uses to prove who it is when it connects to a service, such as an AI model provider.
B
- Backdoor (AI model)
- A hidden behavior planted in an AI model that stays quiet until the model sees a secret trigger chosen by an attacker.
C
- CISA KEV
- A public list from the US Cybersecurity and Infrastructure Security Agency of software flaws that attackers are already using in real attacks.
- Content filter
- A check that reads what goes into or comes out of an AI model and blocks or flags anything that breaks set rules.
- Context window
- The amount of text an AI model can take into account at once, including its instructions, the conversation and any documents you add.
- CVE
- A public ID number given to one specific security flaw, so everyone can refer to the same problem by the same name.
- CVSS
- The standard score, from 0.0 to 10.0, that rates how severe a security flaw in software is.
D
- Data poisoning
- An attack that tampers with the data an AI model learns from or looks things up in, so the model behaves the way an attacker wants.
- Data processing agreement
- A contract between your organization and a service provider that sets out what the provider may do with personal data and how it protects it.
- Data residency
- The physical location, such as the EU or the US, where an organization's data is stored and processed by a service.
- Data retention
- The length of time a service keeps the chats, files and recordings you give it, and what happens to them when you delete them.
- Deepfake
- Realistic fake audio, video or images of a real person, made with AI to show them saying or doing something they never did.
- De-identification
- The process of removing or replacing details in data, such as names, so the records no longer point to specific people.
- Differential privacy
- A mathematical technique that adds measured randomness to data or results so no one can tell whether a specific person's data was included.
E
- Embedding
- A list of numbers that captures the meaning of text, an image or other data, so that similar items end up with similar numbers.
- EU AI Act
- The European Union's law that sorts uses of artificial intelligence by risk, banning some uses and setting duties for others.
- Excessive agency
- The problem of giving an AI tool more power, permissions or independence than its task needs, so a mistake can cause real harm.
F
- Fine-tuning
- Extra training that adapts an existing AI model to a specific task or style using a smaller, focused set of examples.
- Foundation model
- A large AI model trained on broad data that can be adapted to many tasks, often serving as the base for chat assistants.
G
- GDPR
- The European Union law that sets rules for how organizations collect, use and protect personal data about people in the EU.
- GitHub Security Advisory
- A public warning in GitHub's advisory database that names a security flaw in a software package and the versions that fix it.
- Grounding
- The practice of tying an AI model's answers to trusted source material you supply, so its replies reflect those documents.
- Guardrails
- Controls placed around an AI model to keep what goes in and what comes out within set limits, such as filters, rules and approval steps.
H
- Hallucination
- An AI model's false or made-up answer, stated as if it were true, such as an invented fact, quote, citation or number.
I
- Indirect prompt injection
- An attack that hides instructions in content an AI tool reads, like a web page or email, so the tool follows the attacker.
- Inference
- The step where a trained AI model is run to produce an answer, as opposed to the earlier step of training it.
- Inference server
- Software that loads AI models and answers requests to them from other programs or people over a network.
- Insecure deserialization
- A flaw where software unpacks saved or received data without checking it, so a booby-trapped file can run code or change what the software does.
- Insecure output handling
- A mistake where software passes an AI model's answers to other systems, like web pages or emails, without checking them first.
- ISO 27001
- An international standard for how an organization manages information security risks through policies, controls and regular review.
- ISO 42001
- An international standard for AI management systems that organizations, including AI vendors, can be certified against.
J
- Jailbreak
- An attempt to trick an AI model into breaking its own safety rules, such as producing content it is meant to refuse.
L
- Large language model
- An AI model trained on huge amounts of text to predict and write language, used for chat, writing, summaries, translation and code.
M
- Membership inference
- An attack that tries to work out whether a specific person's record was in the data used to train an AI model.
- MITRE ATLAS
- A public catalog of the ways attackers go after AI systems, with real cases and defenses, published by The MITRE Corporation.
- Model card
- A short document published with an AI model that explains what it is for, how it was tested, and where it falls short.
- Model Context Protocol
- An open standard that lets AI assistants connect to outside tools and data through small add-on programs called servers.
- Model denial of service
- A problem where an AI system gets so many costly requests or huge inputs that it slows down, fails or runs up large bills.
- Model evaluation
- The testing of an AI model against set tasks to measure how accurate, safe, fair or reliable it is.
- Model inversion
- An attack that uses an AI model's answers to work out private details about the people or records it was trained on.
- Model weights
- The numbers inside a trained AI model that store what it learned; whoever has them can run the model.
- Multimodal model
- An AI model that can take in or produce more than one kind of data, such as text, images, audio or video.
N
- NIST AI RMF
- A free, voluntary guide from the US government's standards agency for spotting and managing the risks that come with using AI.
- NVD
- The US government's public database of known software security flaws, run by NIST, with added details like how serious each flaw is.
O
- Open-weight model
- An AI model whose inner settings are published, so anyone can download a copy and run it on their own computers.
- Overreliance
- The habit of trusting AI output more than it deserves, so mistakes slip into decisions, documents or code unnoticed.
- OWASP Top 10 for LLM Applications
- A list of the 10 most important security risks for apps built on large language models, published by OWASP.
P
- Path traversal
- A flaw that lets an attacker use tricky file paths to reach files outside the folder an app was meant to allow.
- Personally identifiable information
- Information that identifies a person, either on its own, like a name or email address, or when combined with other details.
- Prompt injection
- An attack where someone writes text that makes an AI model ignore its instructions and follow theirs instead.
R
- Red teaming
- The practice of deliberately trying to make an AI system fail or misbehave, the way an attacker or careless user might.
- Remote code execution
- A security flaw that lets an attacker run their own commands or programs on someone else's computer or server over a network.
- Responsible AI
- An approach to building and using AI that pays attention to fairness, safety, privacy, transparency and accountability.
- Retrieval-augmented generation
- A way of building AI assistants that first searches your documents for relevant passages, then has the AI model answer from them.
- RLHF
- A training method in which people rate an AI model's answers and the model is then adjusted to give the kinds of answers people prefer.
S
- Safety classifier
- A model trained to spot harmful or rule-breaking content, used to screen what people send to an AI tool and what it sends back.
- Sensitive information disclosure
- A security problem where an AI system reveals information it should not, such as personal data, private records, logins or its own instructions.
- Server-side request forgery
- A security flaw that tricks a server into sending web requests for an attacker, often to internal systems the attacker cannot reach.
- Shadow AI
- AI tools that staff use for work without the organization's knowledge or approval, often through free personal accounts.
- SOC 2
- An independent auditor's report on how well a service company protects the systems and information it handles for its customers.
- Synthetic data
- Artificial data made to look like real data in its patterns, so you can test, train or share without exposing real records.
- System prompt
- The hidden instructions an AI tool gives its model before your message, setting its role, tone, rules and limits.
T
- Token
- The small unit of text an AI language model reads and writes, often a whole word or a piece of one.
- Tool use
- The ability of an AI model to reach outside tools, like a search engine or a calendar, while it works out an answer.
- Training data extraction
- An attack that gets an AI model to repeat pieces of the data it was trained on, such as personal details or passages of text.
- Training opt-out
- A setting or request that stops an AI vendor from using what you type or upload to train or improve its models.
V
- Vector database
- A database that stores your content as lists of numbers so an AI tool can find the passages closest in meaning to a question.
W
- Watermarking
- A visible or hidden signal added to AI-made images, audio, video or text to show that it was produced by an AI model.
Z
- Zero data retention
- An arrangement in which an AI provider does not keep your prompts or its answers after it has finished responding to a request.
- Zero-day
- A security flaw that attackers know about or use before the software maker has a fix ready, leaving no time to patch.