OpenAI introduces GPT-4

Background

GPT-4 belongs to OpenAI's GPT line of models. Most modern language models, including these, are based on the Transformer, a neural network architecture that uses attention to relate each part of a text to the other parts that matter for it. OpenAI described GPT-4 as the latest milestone in its effort to scale up deep learning, meaning to build larger models trained on more data and computation.^[1]

What happened

On 14 March 2023 OpenAI announced GPT-4.^[1] The GPT-4 Technical Report was posted to arXiv the following day, 15 March 2023.^[2]

GPT-4 is a large multimodal model that accepts image and text inputs and produces text outputs. Multimodal means it handles more than one kind of input, here images as well as text.^[1] It was made available to paid ChatGPT subscribers and through the API.^[4] The same day, Microsoft said that its Bing AI chatbot used GPT-4, and OpenAI said it had trained the model on Microsoft Azure.^[4][2]

Key details

OpenAI said GPT-4 is still less capable than humans in many real-world scenarios, but that it shows human-level performance on various professional and academic benchmarks. On a simulated bar exam it scored around the top 10% of test takers.^[2]

GPT-4 is a Transformer-style model pre-trained to predict the next token, using publicly available data and data licensed from third parties. It was then fine-tuned with reinforcement learning from human feedback (RLHF).^[2] Pre-training is the first stage, in which a model learns general patterns from large datasets before any task-specific tuning.

Citing the competitive landscape and safety implications, the report gave no further details about the architecture (including model size), the hardware, the training compute, the dataset construction or the training method.^[2] Readers therefore could not check the model's size or design from the report itself.

OpenAI said it had predicted some aspects of GPT-4's performance from models trained with no more than 1/1,000th of GPT-4's compute.^[2] This is a form of scaling prediction, estimating how a large model will perform from smaller training runs.

On translated versions of the MMLU benchmark, a set of multiple-choice questions across many academic subjects, GPT-4 surpassed the previous English-language state of the art in 24 of the 26 languages tested.^[2]

CNBC reported API pricing of about 3 cents for roughly 750 words of prompt and 6 cents for roughly 750 words of response.^[4]

Impact

The release showed that a large language model could take images as well as text, widening the range of tasks it could be used for.^[1] Because GPT-4 was offered both to paid ChatGPT subscribers and through Microsoft's Bing chatbot, the same model served OpenAI's own product and a partner's product at once.^[4]

The technical report withheld architecture and training details. This raised a question that would recur across the industry: how much a developer should disclose about its most capable models, and how outside researchers can evaluate a system whose design is not public. The report's stated reasons were competition and safety. The question of how much outsiders can verify remained open.