AI Security
LLM security: how your AI assistant can be turned against you, and how to stop it
LLM security is protecting a large language model, and everything connected to it, from being tricked, leaking data or acting beyond what anyone intended. The model is rarely the weak point. The risk sits in what it can read, what it can do and how far its answers are trusted.

Published 6 min read
LLM security is protecting a large language model, and everything connected to it, from being tricked, leaking data or acting beyond what anyone intended. The model itself is rarely the weak point. The risk sits in what it can read, what it can do and how far its answers are trusted. The main risks are prompt injection, data leakage, secrets in the prompt, excessive agency and output nobody checks. Each one has a fix you can test before launch, and the incidents below show what happens when nobody does.
Four times an AI was turned against its owner
The $1 SUV. In December 2023 a visitor to a Chevrolet dealer's website told its chatbot to agree with anything the customer said and to end every answer by calling it a legally binding offer. The chatbot then "agreed" to sell a new 2024 Tahoe for $1, as GM Authority reported. The dealer did not honour it and took the chatbot down. Nobody hacked anything. They typed.
The refund that did not exist. Air Canada's website chatbot told a grieving customer that a bereavement fare could be claimed after travelling, which the airline's policy did not allow. In Moffatt v. Air Canada (February 2024), the airline argued the chatbot was responsible for its own words. The tribunal disagreed and ordered the airline to pay.
The code that walked out. In 2023 Samsung engineers pasted internal source code into ChatGPT to get help with it, and the company restricted staff use of generative AI tools, as Bloomberg reported.
The email nobody opened. In June 2025 researchers showed that a crafted email could make Microsoft 365 Copilot pull internal data into an answer and send it out, with no click from anyone (The Hacker News). Microsoft fixed the flaw, CVE-2025-32711, before it was known to be used.
These are not exotic attacks. They are the ordinary ways an LLM fails once real people and real data reach it.
Prompt injection: anyone can give your AI orders
An LLM cannot reliably tell the instructions you gave it from instructions hidden in the text it reads. Prompt injection uses that. In the direct form, a user types the orders, as with the $1 SUV. In the indirect form, the orders hide in an email, a document or a web page the model is asked to read, as in the Copilot case, and the person using the assistant never sees them.
Two mistakes make it worse. Teams treat a line in the system prompt, such as "never reveal this" or "only discuss our products", as a control. It is a request, and a determined user can talk the model out of it. And teams test only what users type into the chat box, not the emails, files and pages the model reads. For the attack itself, see how prompt injection works.
The fix is to assume injection will succeed sometimes, and limit what a tricked model can reach and do. The sections below are where that happens.
AI data leakage: what goes in can come out
Data leaks through AI in four ways we see again and again. Staff paste confidential data, code or customer records into public AI tools nobody approved. An internal assistant returns another team's or another customer's documents, because its retrieval ignores the permissions of the person asking. Prompts and answers end up in logs, fine-tuning sets or evaluation data that are kept and shared too widely. And a vendor's terms allow it to keep or train on what you send, and nobody read them.
The figures say this is common. In IBM's 2025 data breach research, 13% of organisations reported a breach of an AI model or application, and 97% of those had no AI access controls in place. One in five reported a breach involving shadow AI, tools staff adopted on their own, and where shadow AI was widespread the breach cost $670,000 more on average.
The fixes are ordinary data and application security applied to a new channel: approved tools and a clear policy for staff, retrieval that checks each user's access to each document, sensitive data kept out of logs and training sets, and vendor terms read before the contract is signed.
Secrets in the prompt
The system prompt, or the code around it, often holds things it should not: API keys, internal addresses, discount rules, the names of other systems. Secrets also reach the model by accident, pasted into the context or sitting in a document it retrieves. A model will repeat whatever is in its context if asked the right way, so anything in the prompt should be treated as visible to every user.
Keep keys in a secrets manager and call other systems from code, never from the prompt. Enforce business rules, such as who gets a discount, in code that checks the model's output, not in a sentence asking the model to behave.
Excessive agency: when your AI can act on its own
An assistant that can only answer is embarrassing when it is tricked. One that can send email, issue refunds, change records or delete files is dangerous. The problem we find most often is not a clever attack but broad permissions: an agent given a full mailbox when it needs one folder, or an admin account when it needs to read one table, and no person between its decision and the action.
Give each tool the narrowest permission the task needs, and require a person to approve anything that sends, pays, changes or deletes. If the $1 SUV chatbot had been connected to the dealer's ordering system, that approval step would have been the only thing standing between a joke and a sale.
Output nobody checks
What the model writes is often trusted as if a developer had written it: shown as HTML on a page, run as code or a database query, or passed to another system. A tricked model can then carry an attack into those systems. Even without an attacker, the answer can be wrong, and as Air Canada learned, a company answers for what its chatbot tells customers.
Treat model output like any other untrusted input. Validate and encode it before it is displayed, executed or sent onward, and ground customer-facing answers in approved content, with a hand-off to a person when the question goes beyond it.
Why "the vendor handles it" is not a plan
The model provider secures the model and its platform. How you connect it is yours: the data it can retrieve, the tools it can call, the secrets around it, who can reach it and what happens to its answers. Most problems we find sit in that setup. The other gap is visibility. If prompts, retrieved documents and tool calls are not logged, a leak or an abuse leaves no trace, and nobody knows to respond.
AI security best practices before go-live
Before an LLM reaches real users, we fix four things first:
- Least privilege. Cut data and tool permissions to what the task needs, and make retrieval respect each user's access.
- Human approval. A person approves every action that sends, pays, changes or deletes.
- Checked output. Validate and encode model output before it is displayed, executed or sent onward.
- Limits and logs. Rate and cost limits per user, and logs of prompts, retrievals and tool calls, with alerts your team can act on. Our threat detection and SIEM service writes those rules into the platform your team already runs.
For the full list of LLM vulnerabilities, the OWASP Top 10 for LLM applications is the reference most teams start from; its 2026 edition, released in August 2026, covers each risk above, with secrets in the prompt now under hidden context exposure. NIST's generative AI profile of its AI risk management framework places them in a wider governance process.
Then test it the way it will be attacked, through the chat box and through everything the model reads. Our AI system security review and hardening does exactly that for the systems you build or buy: a review of one system in two to six weeks at a fixed price, then the fixes made alongside your engineers and tested again.
Frequently asked questions
Is LLM security different from application security?
It builds on it. Access control, input checks and logging still matter. What changes is that the model takes orders from any text it reads, so the line between data and instructions disappears, and an assistant connected to tools can act on those orders. LLM security is mostly about limiting what the model can reach and do, and checking what it produces.
Doesn't the model provider handle security?
The provider secures the model and its own platform. How you connect it is yours to secure, that is, the data it can retrieve, the tools it can call, the secrets around it, who can reach it and what happens to its answers. Most of the problems we find sit in that setup, not in the model.
What is AI data leakage?
Sensitive data leaving the company through an AI system. It happens when staff paste confidential material into tools nobody approved, when an internal assistant returns documents the person asking should not see, when prompts and outputs are logged or reused for training without controls, or when a vendor's terms allow it to keep what you send.
How do you test an LLM before it goes live?
Map what it can read, what it can call and who can reach it, then attack it the way a user or an outsider would, through the chat box and through every document, email or web page it reads. Our AI system security review does this for one system in two to six weeks, at a fixed price, and reports each finding with how to fix it.
Sources
- Cost of a Data Breach Report 2025, press release, IBM, 2025
- OWASP Top 10 for LLM Applications 2026, OWASP GenAI Security Project, 2026
- NIST AI 600-1, AI RMF: Generative Artificial Intelligence Profile, NIST, 2024
- Moffatt v. Air Canada, 2024 BCCRT 149, Civil Resolution Tribunal of British Columbia, 2024
- GM dealer chat bot agrees to sell 2024 Chevy Tahoe for $1, GM Authority, 2023
- Samsung bans staff's AI use after spotting ChatGPT data leak, Bloomberg, 2023
- Zero-click AI vulnerability exposes Microsoft 365 Copilot data without user interaction, The Hacker News, 2025
AI governance consulting and AI security
We help you govern, secure and watch the AI systems your company builds or buys. That means a governance framework your leadership can stand behind, a security review that tests how each system can be misused, the hardening it needs, and detection your team can operate.
Explore AI SecurityRelated reading
AI Security
10 ways your AI can be turned against you: OWASP Top 10 for LLMs vs MITRE ATLAS
A chatbot that invents a refund policy, an assistant that leaks mail without a click, a coding tool shipped with planted code. The OWASP Top 10 for LLM Applications names these risks for builders; MITRE ATLAS records how attackers carry them out. Here is how the two fit together.
7 min read
AI Security
Agentic AI security: your newest employee has admin rights and no manager
AI agents do more than answer. They call tools, read data and act in other systems, often with a person's keys. Five public incidents from 2025 show what goes wrong, and each points to a control, from an identity of the agent's own to a named owner who can stop it.
7 min read
Tell us which AI systems you need to govern
List the tools and systems you use, built or bought. We reply within two business days.