ESM3

A single model for three kinds of protein data

ESM3 is a generative model for proteins, announced by EvolutionaryScale on 25 June 2024. The company introduced it as the first generative model for biology that can reason over protein sequence, structure and function at the same time.^[1] A protein can be described in several ways: as a chain of amino acids, as a three-dimensional shape, or as a statement of what it does in a cell. ESM3 is built to handle those descriptions together rather than one at a time.

Three alphabets, one training objective

To train a single model on all three kinds of data, structure and function are converted into discrete alphabets, so that sequence, structure and function can all be learned with a masked-token objective: parts of the input are hidden, and the model is trained to recover them.^[1] The flagship model has 98 billion parameters and was trained with over 1e24 FLOPs, which EvolutionaryScale called the most compute ever applied to training a biological model.^[1] The training set was also augmented with hundreds of millions of synthetic data points, including predicted structures and functions.^[1]

Generation runs the same process in reverse. The model starts from fully masked tokens and unmasks them iteratively, and each step can be guided by any combination of sequence, structure or function prompts.^[1] In practice, a user can supply a desired shape, a description of what the protein should do, or part of a sequence, and let the model fill in the rest.

The fluorescent protein experiment

Fluorescent proteins are widely used in biology as markers: they glow, and can be attached to other molecules to make them visible under a microscope. In one experiment ESM3 was used to generate a new one, esmGFP. Its sequence is only 58 percent identical to the closest known fluorescent protein.^[1][2] The team estimates that this sequence distance is equivalent to simulating over 500 million years of evolution.^[1][2]

That first experiment did not hand back a finished, natural-quality protein. Of 96 generations, several fluoresced, and one of them, called B8, was 50 times dimmer than natural GFPs.^[1] In other words, the model produced proteins that genuinely glowed, but far more weakly than the natural versions they were compared against.

The open checkpoint and its safeguards

Alongside the larger models, an open checkpoint was released. It was trained on 2.78 billion natural proteins, augmented to 3.15 billion sequences, 236 million structures and 539 million function annotations, or 771 billion tokens in total.^[3] Virus-related sequences and proteins on the USDA Select Agents and Toxins list were removed from the training data, and the function decoder was filtered for harmful keywords.^[3] At launch the model came in small, medium and large sizes, with the open model released under a non-commercial licence.^[4]

Where it sits

A peer-reviewed version of the work appeared in Science on 16 January 2025.^[2] The interest in a model like ESM3 comes from the shape of protein design as a problem. The number of possible amino acid sequences is enormous, laboratory experiments are slow, and evolution has explored only a small part of that space. A model that can be pointed at a desired structure or function offers a way to propose candidates before anything is made at the bench. The fluorescent protein result shows both sides of that: a sequence very far from anything known that still worked, and a first-generation protein far dimmer than what nature already provides.