Preferred Networks releases its first large language model, PLaMo-13B

Background

The "weights" of a large language model are the set of numerical parameters it learns during training. Models are usually released in one of two shapes: behind an interface, where users call the model remotely, or as downloadable weights that anyone can run on hardware they control. The second kind is commonly called an open-weight release. What others may do with those weights — above all, whether they may use them commercially — depends on the licence the publisher attaches.

The release

On 25 September 2023, Preferred Networks published PLaMo-13B ^[1]. It was the company's first large language model, and also its first open-weight release ^[1]. The two firsts arrived together: the company had built a model of this kind from scratch, and at the same time handed the resulting weights to outsiders rather than keeping them behind a service.

How it was trained

According to the published material, PLaMo-13B was trained on 1.5 trillion tokens, 88% of them English ^[1]. A token is the unit a model actually processes — roughly a word fragment, so a trillion tokens is an amount of text far beyond what a person could read. The language mix matters because it shapes what the model has seen most often: with English at 88%, the bulk of its exposure during training was English text.

The company accompanied the release with a technical blog post explaining the training setup ^[2]. Publishing that alongside the weights gives outside readers a record of how the model was built, rather than only the finished artefact.

Licence and testing

Preferred Networks chose Apache-2.0 for the release, which makes the weights usable commercially without a separate agreement ^[1]. That is a permissive open-source licence: instead of a case-by-case negotiation or a restriction to research use, the terms are the same for everyone and commercial use is included.

The model card also carries a warning: testing covered only English and Japanese ^[1]. In practice that means the documented evaluation does not extend to other languages, and anyone using the model elsewhere is working without a published record of how it behaves.

The instruction-tuned follow-up

An instruction-tuned version appeared a month later, on 26 October 2023 ^[1]. Instruction tuning is a second training stage in which a model is shown examples of requests paired with the responses that are wanted, so that it learns to follow a request rather than merely continue text. A base model that has only been trained to predict the next token can be awkward to use directly; the tuned version is meant to be asked things and answer them.

Why it matters

The release marks a starting point for the company's language-model work: the first model of this kind it built, and the first time it published weights at all ^[1]. The licence choice lowers the barrier for anyone wanting to try the model in a commercial setting, since no separate agreement has to be arranged first ^[1]. And the combination of an open licence, a technical blog post describing the training setup ^[2], and a model card that states the limits of testing ^[1] leaves a written record that later readers can check against what was actually claimed at the time.