Jailbreak

What a jailbreak is

A jailbreak is an adversarial prompt that makes a safety-trained model produce behaviour its training was meant to prevent.^[1] The term is used for inputs designed to get around guardrails, not for ordinary mistakes or capability limits. The target is a model that has already been trained to refuse or avoid certain kinds of output.

This matters because safety training is often treated as a layer that can be added to a capable model. A jailbreak shows that layer can fail under inputs chosen to exploit it.

Two failure modes

In July 2023, Wei et al. published an analysis of why such attacks succeed. They noted that large language models trained for safety and harmlessness remain susceptible to adversarial misuse.^[1] Rather than treating each attack as an isolated trick, they proposed two failure modes of safety training: competing objectives and mismatched generalization.^[1]

Competing objectives arise when a model's capabilities and its safety goals conflict.^[1] A model trained to be helpful, for example, may follow a request framed as a legitimate task even when the underlying content is harmful; the helpfulness objective and the safety objective pull in different directions. This is a hypothesis about why safety training can be bypassed, not a claim that the model has a single explicit goal.

Mismatched generalization occurs when safety training fails to generalize to a domain in which the model has capabilities.^[1] Safety training may cover some formats, languages, or styles, while the model's general capabilities extend much further. An input that looks unlike the safety training data can therefore reach behaviour the training was meant to suppress.

Testing the explanation

The authors used those failure modes to guide the design of new attacks and evaluated state-of-the-art models, including GPT-4 and Claude v1.3, against existing and newly designed attacks.^[1] They found that vulnerabilities persist despite the extensive red-teaming and safety training behind those models.^[1]

The new attacks, built on the two failure modes, succeeded on every prompt in a collection of unsafe requests drawn from the models' own red-teaming evaluation sets, and outperformed existing ad hoc jailbreaks.^[1] That result is significant because the test prompts came from the models' own evaluation material, not from a narrow set of hand-written examples.

Why it matters

The paper argues for safety-capability parity: safety mechanisms should be as sophisticated as the underlying model.^[1] It also argues against the idea that scaling alone can resolve these failures.^[1] In other words, a bigger or more capable model does not automatically make safety training more robust; the safety side has to keep pace with the capability side.

For readers, the concept is a reminder that safety training is not a one-time fix. Jailbreak research is adversarial: attacks are designed to find the gap between what a model can do and what its safety training covers. The two failure modes give a way to organise that search, and they point to the kind of defence that would be needed — safety mechanisms that generalise as broadly as the model's capabilities and that do not collapse when objectives conflict.^[1]