Security & Policy
Prompt Injection
An attack that hides malicious instructions in content a model reads, tricking it into ignoring its real orders.
Prompt injection exploits the fact that a model cannot always tell its trusted instructions apart from the text it is processing. An attacker plants commands in a web page, email, or document the model ingests, and the model may follow them, leaking data or taking unwanted actions. It is widely considered the top security risk for LLM applications, and it is especially dangerous for agents that can act on the world, with no fully reliable fix yet.
Reference: Wikipedia: Prompt Injection
Related terms
Jailbreak
A prompt crafted to trick a model into bypassing its own safety rules and doing something it should refuse.
Guardrails
Controls placed around a model to keep its inputs and outputs safe, on-topic, and within policy.
AI Agent
An AI system that can take actions and use tools to pursue a goal, not just answer a single question.
Red Teaming
Deliberately probing an AI system to find ways it can fail, be misused, or produce harmful output, before attackers do.