EvoLLM-JP

What it is

EvoLLM-JP is a Japanese language model released by Sakana AI. The 7B version was uploaded to Hugging Face on 6 March 2024, before the paper describing its method appeared^[2]. Sakana describes it as an experimental general-purpose Japanese model^[2].

What makes it unusual is not its size but how it was made: it was not trained from scratch, but assembled by merging existing open-source models. The merged model has mathematics ability that none of the three source models had on its own^[1][2].

Where it came from

The 7B model was produced by merging Shisa Gamma 7B v1, WizardMath 7B v1.1 and Abel 7B 002^[2]. The method behind it, published by Sakana AI as the paper "Evolutionary Optimization of Model Merging Recipes" on 19 March 2024^[1], came after the model itself, so the order was: model first, paper second^[1][2].

The paper's approach drops the usual practice of hand-picking which models to combine and in what proportions. Instead, an evolutionary search looks through a large space of possible merges and finds good ones automatically^[1]. The search runs in two spaces at once: parameter space and data-flow space. The first governs how weights are blended; the second allows layers to be rewired, so a candidate is not just a weighted average of several models but can also change how data flows through the network^[1].

How to think about this kind of merging

Model merging, in general, takes several models that have already been trained and combines them into one, without collecting new data or rerunning a large training job. The difficulty is that the number of possible combinations is enormous: which models to pick, how much weight each should get, which layer should line up with which. Trying these by hand quickly becomes impractical. Evolutionary search treats each combination as a candidate, evaluates it, keeps the better ones, and produces new candidates by modifying them, repeating the cycle. Because the search covers data flow as well as weights, modifications can land on how layers are connected rather than only on the numbers being averaged^[1]. The attraction is that no retraining is needed; the cost is that the outcome depends heavily on how candidates are generated and evaluated.

Other results from the same recipe

EvoLLM-JP is not a one-off. A 10B variant was uploaded on 8 March 2024^[2], and earlier, on 4 March 2024, the same recipe produced EvoVLM-JP, a Japanese vision-language model^[2].

On licensing, the 7B model uses a custom licence, recorded on Hugging Face as "other" rather than as an OSI licence^[2]. Anyone building on it needs to read the terms first.

Why it matters

Sakana released the method as a general recipe for cross-domain merging, not something restricted to Japanese models^[1]. EvoLLM-JP is therefore best read as a demonstration: the source models were each limited in their own way, yet the combination found by search exceeded every one of them in a particular direction^[1][2]. It also turns the question of how to combine models into something that can be searched automatically rather than settled by hand^[1].

^[1]: arXiv, "Evolutionary Optimization of Model Merging Recipes" ^[2]: Hugging Face, "EvoLLM-JP-v1-7B model card"