Program Synthesis with Large Language Models (MBPP)

The question the paper takes on

Much of everyday programming consists of small, well-specified jobs: read a list of numbers, sort it, return the second largest. The description is in ordinary language; the result is a few lines of Python. Can a language model do that translation? The paper, submitted to arXiv on 16 August 2021 by Jacob Austin, Augustus Odena, Maxwell Nye and eight other authors, asks how far this ability reaches and how it changes as models grow.^[1]

Two new benchmarks, a wide range of model sizes

To answer the question the authors built test sets rather than relying on existing ones. They evaluate models with between 244M and 137B parameters on two new benchmarks, MBPP and MathQA-Python, and they do so in two regimes: few-shot learning, where the model is given a handful of examples in the prompt and must continue the pattern, and fine-tuning, where the model's weights are further trained on task data.^[1]

MBPP — Mostly Basic Programming Problems — contains 974 programming tasks designed to be solvable by entry-level programmers. Each task pairs a natural-language description with the short Python program that should satisfy it, so the benchmark measures the ability to synthesize a program from a description rather than to recall facts about code.^[1] Because the tasks are small and self-contained, a solution can be checked automatically by running it against test cases, which is what makes a benchmark of this kind practical to score at scale.

The few-shot/fine-tuning split matters because the two ask different things. Few-shot results say something about what a model can do out of the box, with no code-specific training and only a prompt to work from; fine-tuning results say how much of the remaining gap can be closed with a modest amount of supervision.

What the models could do

The headline result is about scale. The largest models, even without fine-tuning on a code dataset, can synthesize solutions to 59.6 percent of MBPP problems using few-shot learning with a well-designed prompt. Fine-tuning on a held-out portion of the data improves performance by about 10 percentage points across most model sizes.^[1] The gain from fine-tuning is therefore roughly the same whether the model is small or large, while the starting point differs enormously.

On both datasets, synthesis performance scales log-linearly with model size.^[1] In plain terms, each multiplicative increase in parameters buys an additive improvement in the share of problems solved: the curve rises steadily rather than flattening out or jumping unpredictably, which is what makes it possible to talk about a scaling trend at all.

The paper also looks at what happens after the model has produced an answer. When a human gives natural-language feedback on the model's initial attempt, the error rate of the revised attempt is halved compared with the initial prediction.^[1] This is a different use of language than generation: the description of what went wrong, in words rather than as a corrected program, carries enough information to fix a large share of the failures.

Why it matters

Program synthesis — getting a machine to produce a program from a specification — has a long history, but the interesting claim here is about measurement. By pairing a large set of small, automatically checkable tasks with a sweep across two orders of magnitude of model size, the paper turns a vague question ("can models write code?") into a curve with a slope. The 59.6 percent figure and the roughly 10-point fine-tuning gain are useful precisely because they come with the conditions attached: which prompt, which regime, which model sizes.

The feedback result points in a second direction. If a plain-language comment from a person can halve the error rate, then the model is not only generating code but can act on criticism of it, which suggests a workflow in which generation and correction alternate rather than a single one-shot answer. The benchmark and the scaling measurements together give later work a common yardstick: a fixed set of short tasks, a stated prompting setup, and results reported as a function of model size.^[1]