GPT-4 Technical Report

What the report is

The GPT-4 Technical Report is a paper credited to OpenAI, submitted to arXiv on 15 March 2023 with the identifier 2303.08774^[1]. Its abstract describes GPT-4 as a large-scale, multimodal model that can accept image and text inputs and produce text outputs^[1]. It is a technical document rather than a product announcement: it sets out what the model is, how it was trained, what it was measured on, and how the engineering work around it was organised.

How the model is built

GPT-4 is described as a Transformer-based model pre-trained to predict the next token in a document^[1]. Next-token prediction is a simple objective with a demanding side effect. A document is split into tokens, and at every position the model produces a probability for each token that might come next. Training adjusts the model's parameters so that the token that actually appears gets a higher probability. Doing this at scale over large amounts of text forces the model to represent grammar, facts and something like reasoning, because all of these help predict what comes next. The Transformer is the network family that makes this practical: each position in the sequence can draw on information from other positions when building its representation, and the report places GPT-4 within that family^[1].

Inputs are not text alone. The abstract says the model accepts image and text inputs while producing text outputs^[1], so images can serve as part of the question even though the answer is always written text.

Pre-training is followed by a post-training alignment process, which the report says improves performance on measures of factuality and on adherence to desired behaviour^[1]. Alignment here means further training after the main pre-training run, aimed at shaping how the model responds: making its statements more reliable and keeping its behaviour within intended bounds.

What it was measured on

The report is explicit about the ceiling as well as the floor. It states that the model is less capable than humans in many real-world scenarios, but that it exhibits human-level performance on various professional and academic benchmarks^[1]. The example given in the abstract is a simulated bar exam, on which the model scored around the top 10 percent of test takers^[1]. The bar exam is a licensing examination for lawyers, built from written questions and essay-style problems; a simulated version is a practice paper rather than the official test. The two halves of that sentence are meant to be read together: benchmarks are narrow, scored tasks, and doing well on one is not the same as matching people across the messy situations of ordinary work.

Predictable scaling as an engineering goal

The report calls the development of infrastructure and optimisation methods that behave predictably across a wide range of scales a core component of the project^[1]. That work allowed the team to predict some aspects of GPT-4's performance from models trained with no more than one thousandth of its compute^[1]. The reasoning behind treating this as a deliverable is practical. A full training run is expensive and slow, while a small run is cheap. If performance curves stay smooth and predictable as models grow, then a series of small experiments tells you something about the large model before you commit the resources to train it, and problems can be found early rather than at the end.

The document itself

The report runs to 100 pages, and its author list was updated in later revisions^[1]. That length is unusual for a conference paper and closer to what industrial labs produce when a single document has to cover the model, its evaluation and the engineering around it.

Why it matters

The report is a compact example of how frontier model work came to be described in public. It makes a capability claim and states its limits in the same paragraph, so a reader can see that human-level scores on a benchmark and weaker-than-human behaviour in real situations are reported as facts about the same system^[1]. It also treats predictability across scales as an engineering result in its own right, not merely a property of the finished model^[1]. And because it is written as a technical document with a fixed identifier and a dated submission, it gives later readers a stable reference point for what was claimed about GPT-4 at the time.