Prompt injection

What prompt injection is

Prompt injection is an attack in which text placed in a model's input overrides the instructions the model was originally given, and with them the controls a developer has built around it^[1]. The weakness is structural rather than a defect in one particular product. A language model reads instructions and data as a single stream of text, and nothing in that stream reliably marks where the developer's request ends and the untrusted content begins. When a later sentence conflicts with an earlier instruction, following the later sentence is a plausible outcome rather than an obvious error.

Two routes: direct and indirect

The earlier work on the subject assumed that the user is the one directly prompting the model. The 2023 study by Greshake and co-authors asked a different question: what happens when the prompt does not come from the user at all, but instead arrives inside data the application retrieves^[1]? That case is indirect prompt injection. The adversary plants text in a place the application is likely to fetch, and no direct interface with the system is needed, so the attack can be mounted remotely^[1].

A hypothetical example shows the shape of the route. An assistant is asked to open a web page and write a summary of it. The page carries ordinary text meant for human readers, plus a hidden passage addressed to the assistant that tells it to set the summarizing task aside and instead relay other material it can see. To the model, both passages are the same kind of thing: text in its input. The developer's request to summarize holds no privileged position once a conflicting instruction appears later in the same context.

Where the concept comes from

Greshake and colleagues published the paper that named indirect prompt injection in February 2023^[1]. Their central argument is that applications built around large language models blur the line between data and instructions^[1]. Before that framing, the threat model treated the user as the input source; once an application feeds retrieved content to the model, the input source becomes everything the application can reach.

What the study demonstrated

The authors derive a computer-security taxonomy of impacts, among them data theft, worming and information ecosystem contamination^[1]. Worming here means a payload that gets an application to reproduce it in further content, so that it spreads. These were not only hypotheticals: the authors demonstrated practical viability against real systems, including Bing's GPT-4 powered Chat and code-completion engines, as well as synthetic applications built on GPT-4^[1].

The paper also describes how much the mechanism can do. Processing retrieved prompts can act as arbitrary code execution, manipulate the application's functionality, and control how and whether other APIs are called^[1]. In ordinary software, code execution means an attacker's instructions run on the machine; here what gets steered is the model's own decision-making, and through it the tools the application allows the model to call.

Limits and open problems

The authors concluded that effective mitigations for these threats were lacking at the time of publication^[1]. That conclusion points to the limitation of the concept itself: it names a failure mode and supplies a vocabulary, but it does not come with a fix. Filtering is hard because legitimate content and attacking content travel the same channel; a rule sharp enough to catch text that is instructing the model will also catch ordinary prose, and discarding outside content wholesale removes the reason the application retrieves it. The practical value of understanding prompt injection is not memorizing a particular attack string but recognizing that a system which puts untrusted text and instructions into one input has handed over part of its control.