Seed-OSS-36B

Three models in one release

On 20 August 2025 ByteDance Seed released three open-weight language models under one name: Seed-OSS-36B-Base, Seed-OSS-36B-Base-woSyn and Seed-OSS-36B-Instruct.^[1] The first two are pre-trained base models. The difference between them is that the "woSyn" variant was pre-trained without synthetic instruction data, which lets researchers compare a model trained on ordinary text with one that also saw machine-generated instruction material.^[1] The Instruct model is the version tuned for dialogue and task following, and it is the one whose benchmark results are reported in most detail.^[1] All three are released under the Apache-2.0 licence, which permits commercial and non-commercial use, modification and redistribution.^[1]

Architecture

Seed-OSS-36B is a causal language model — the standard left-to-right, next-token-prediction design — with 36 billion parameters spread over 64 layers.^[1] It uses grouped-query attention (GQA), in which 80 query heads share 8 key/value heads; this keeps the memory cost of the attention cache well below what 80 independent key/value heads would require.^[1] The hidden size is 5,120 and the attention head size is 128, with a vocabulary of 155K tokens.^[1] Positional information comes from rotary position embeddings (RoPE) with a base frequency of 1e7, normalization uses RMSNorm, and the feed-forward blocks use SwiGLU activations.^[1]

Training used 12 trillion tokens, and the model has a native context window of up to 512K tokens — long enough to hold whole books or large code repositories in a single prompt.^[1] The release presents 12 trillion tokens as a comparatively small pre-training budget for a model of this size; the claim being tested is that careful data work can matter as much as raw token count.^[1]

A budget for thinking

The most distinctive feature is a user-set "thinking budget".^[1] ByteDance recommends setting it to multiples of 512 tokens, and a budget of 0 makes the model answer directly without a reasoning trace.^[1] While reasoning, the model emits self-reflection tags that report how many thinking tokens it has already spent and how many remain in its budget.^[1] In effect the caller gets one dial that controls how long the model is allowed to deliberate, and the model itself can see how much of that allowance is left.

For sampling, the model card recommends a temperature of 1.1 and top_p of 0.95.^[1]

Benchmark results

The Instruct model's published scores include AIME24 91.7, AIME25 84.7, LiveCodeBench v6 67.4, MMLU-Pro 82.7, MMLU 87.4, GPQA-D 71.4, IFEval 85.8, MMMLU 78.4 and RULER at 128K 94.6.^[1] On the software-engineering benchmark SWE-bench Verified it scored 56 when paired with the OpenHands scaffold and 47 with the Agentless (4x10) scaffold.^[1] The two numbers for the same model are a reminder that results on agentic coding tasks depend heavily on the surrounding harness — the tools, the retry loop and the way failures are fed back — and not on the model alone.

The base model, in its synthetic-data version, is reported at MMLU-Pro 65.1, MMLU 84.9, GSM8K 90.8, MATH 81.7 and HumanEval 76.8, the kind of profile expected of a pre-trained checkpoint before instruction tuning and long reasoning traces are applied.^[1]

Where it sits

Seed-OSS-36B belongs to the middle of the open-weight range: large enough to handle long-context reading and multi-step reasoning, small enough that its weights are practical for organisations that want to run and fine-tune a model themselves. Its distinguishing choices are the unusual pre-training budget, the 512K native context, the Apache-2.0 licence and the explicit thinking budget, which turns reasoning length into a parameter of the request rather than a fixed property of the model. Packaging a base model and a synthetic-data-free base model alongside the instruct model also gives researchers a controlled pair for studying what synthetic instruction data actually contributes.