PLaMo-13B

What PLaMo-13B is

PLaMo-13B is a large language model published by Preferred Networks. The company uploaded the model to Hugging Face on 25 September 2023 ^[1]. The number in the name gives the scale: 13B means 13 billion parameters. Architecturally it is a decoder-only model built on LLaMA, the openly published design that many labs have used as a starting point. A decoder-only model does one thing over and over: it reads the text so far and predicts the next token, appends that token to the input, and predicts again, which is how a whole answer is produced token by token ^[1]. PLaMo-13B was pre-trained on a mix of English and Japanese open datasets, so it was built to handle both languages rather than English alone ^[1].

What went into training

The pre-training run covered 1.5 trillion tokens, of which 1.32 trillion were English and 0.18 trillion Japanese ^[1]. English therefore made up the bulk of the corpus, with Japanese as a substantial second share. The English side drew on C4, Project Gutenberg, RedPajama and English Wikipedia ^[1]; the Japanese side on mC4 and Japanese Wikipedia ^[1]. In rough terms, C4 and mC4 are cleaned web-crawl collections, with the "m" standing for the multilingual version; Project Gutenberg supplies public-domain books; RedPajama is an open collection assembled for training language models; and Wikipedia provides encyclopedic text. Mixing web text, books and encyclopedia articles is a common way to give a model both everyday language and longer, edited prose.

Text has to be cut into tokens before training, and PLaMo-13B uses a SentencePiece tokenizer trained on a subset of its own pre-training data ^[1]. Fitting the tokenizer to the corpus the model will actually read means the vocabulary matches the languages and the kind of text involved, which matters more when one model has to cover both English and Japanese. The model's context length is 4,096 tokens ^[1] — the amount of text it can hold in view at once, counting both the prompt it is given and the answer it writes.

Licence and release

PLaMo-13B was released under Apache-2.0, a permissive open-source licence that allowed commercial use from the start ^[1]. For a company that publishes model weights, the licence is as consequential as any technical choice: a permissive licence lets other organisations build products on top of the model, fine-tune it on their own data and redistribute it without negotiating separately. Alongside the release, Preferred Networks published a companion post on its technical site describing the model ^[2].

What came next

An instruction-tuned version, PLaMo-13B-Instruct, followed on 26 October 2023 ^[1]. Instruction tuning starts from a model that has learned to continue text and trains it further on examples of instructions paired with good responses, so that it replies to a request instead of merely carrying on whatever text it is given. The roughly one-month gap between the base model and the instruction-tuned one is the usual pattern for open releases: the base model first, then the version meant for direct use.

Where it sits

PLaMo-13B belongs to a group of openly licensed models built with languages other than English as a first-class target rather than an afterthought. At 13 billion parameters it sits in the middle of the range: large enough to hold a broad body of language, small enough that research groups and smaller companies can realistically fine-tune and run it. Its combination of a permissive licence, an open base model and a bilingual English–Japanese corpus makes it useful as a starting point for work in Japanese — domain adaptation, evaluation, or building instruction-following systems on the base weights.