Sakana AI merges open models by evolutionary search to build a Japanese LLM

Background

Model merging is a way to build a new model by combining existing ones rather than training from scratch. In practice, it often relies on human intuition and domain knowledge: a person decides which models to combine and how to set the mixture. Sakana AI's paper, posted on 19 March 2024, describes this dependence as the usual constraint and proposes an alternative.^[1]

What happened

The alternative is evolutionary search over combinations of open models. Instead of hand-picking a recipe, the method searches through candidate combinations. The authors say it needs no extra training data or compute. This changes model merging from a manual design task into a search problem over models that are already available.^[1]

Key details

The authors say the approach goes beyond weight blending. Weight blending combines parameter values; their method also optimises the data flow between layers. The search is therefore not limited to how strongly each model's weights are mixed. It can also change how information passes through the combined network.^[1]

To demonstrate the idea, the team built a Japanese model that also handles mathematics, a cross-domain case. The released 7B model merges Shisa Gamma 7B v1, WizardMath 7B v1.1 and Abel 7B 002.^[2] The weights went up on Hugging Face on 6 March 2024, thirteen days before the paper.^[2] The models are published as experiments rather than products.^[2]

Why it matters

The event is a concrete example of using search to compose open models. It suggests that a cross-domain model can be assembled from existing checkpoints without extra training data or compute, at least in the setting the authors describe.^[1] It also shows why model merging is more than a single weighted average: optimising the data flow between layers opens another dimension for the search.^[1]

The release as an experiment sets expectations. The 7B model is not presented as a product, but as a test of the method.^[2] That framing puts the emphasis on the recipe and the search procedure, not on a finished system. The paper and the model card together document a specific case: a model aimed at Japanese and mathematics, built from named open models, with weights available before the paper appeared.^[1][2]

For readers, the important point is the workflow rather than a single score. The paper describes a way to define a space of model combinations and search it; the model card records what came out of one such run. The work therefore sits at the meeting point of model merging and automated search, and it makes that meeting point visible through a released model.^[1][2]