Why AI Security is Different

Every piece of software has a security surface. Web apps get SQL injected. APIs get abused. Databases leak. But AI systems, especially LLM-powered agents, add a category of attack that traditional software does not have. This lesson explains why the old playbook is not enough.

The Instruction-Data Boundary Is Blurred

In traditional software, code and data are separate. Your database stores rows of data. Your code reads those rows and does something with them. The rows themselves cannot instruct the code to behave differently. SQL injection is dangerous precisely because it violates that separation: attacker-supplied data is treated as SQL code.

LLMs erase the boundary from the start. An LLM reads its entire input as one continuous stream of tokens. There is no built-in mechanism that says "these tokens are instructions from the developer, those tokens are data from the user, and this third block is untrusted content from a scraped web page". The model does its best to figure out which is which. Attackers exploit this.

When your agent fetches an email and pipes it into the model, the email's content can contain instructions that the model will follow. Not because the model is broken, but because from the model's perspective the email is just more tokens, and tokens are what it reasons over.

Every Deployed Model Is An Attack Surface

If your product accepts user input and sends it to an LLM, you have a target. If your agent has access to tools, those tools are reachable through the model. If your MCP server exposes data, that data is fetchable through any client the user has connected.

The larger the surface, the more places attackers can try. A chatbot that only replies is one surface. A chatbot with a database tool is two. A chatbot with database plus email plus web fetching plus calendar is five. Every added tool is a new door.

What Has Actually Happened

Real 2024-2026 incidents that shaped the discipline:

• Mid-2025: an open-source MCP server for GitHub had a search tool that fetched issue bodies. Attackers filed issues with embedded prompt injections instructing the model to exfiltrate repository secrets. Multiple production teams had secrets leaked before anyone connected the dots.
• Late 2025: a customer-support agent for a major SaaS product had a refund tool with no per-customer limits. A stuck agent (not a malicious attacker) issued hundreds of duplicate refunds over one weekend. Company loss estimated at over R2 million before it was noticed.
• Early 2026: a project-management MCP server had a "create task" tool with no rate limit. A misbehaving client created 40,000 duplicate tasks in an afternoon.
• Mid-2026: a filesystem MCP server marketed as read-only had a bug that allowed writes to files matching a specific pattern. Attackers used it to plant malicious code in dependencies of projects opened by MCP-connected coding assistants.

None of these were exotic. Every one was a straightforward failure to think about "what if the model does something I did not intend, at scale".

The Three Categories You Need to Understand

1. Prompt injection (direct and indirect): attacker gets the model to follow instructions the developer did not intend. Direct is a user typing "ignore your rules". Indirect is a document, web page or database record containing hostile instructions that the model reads as context. Covered in the next lesson.

2. Jailbreak (subset of prompt injection focused on safety bypass): getting the model to produce content or take actions that its safety training was supposed to prevent. Roleplay bypasses, hypothetical framings, encoding tricks. Covered in lesson 3.

3. Tool abuse: the model is manipulated (or gets stuck) into calling tools in ways that harm the user, the service or third parties. SSRF via HTTP tools, path traversal via file tools, amplification via paid APIs, destructive actions without confirmation. Covered in lesson 4.

The Defender's Advantage (Small)

You are not defenceless. You control the system prompt, the tools you expose, the arguments they accept, the confirmations you require and the observability you build in. Every category above has established defences. The rest of this module goes through them one at a time.

The disadvantage is that safety training in the base model helps but is not sufficient. You cannot outsource security to Anthropic or OpenAI. Your product's threat model is your responsibility.

Key Takeaway

AI security is not "regular security plus some prompts". LLMs blur the code/data boundary from the ground up, every deployed model is a new attack surface, and real 2025-2026 incidents show that basic hygiene (input scoping, rate limits, tool discipline, human-in-the-loop) prevents almost all of them. The next four lessons walk through the specific defences.