Llama-3.1-Nemotron-70B-Instruct

What it is

Llama-3.1-Nemotron-70B-Instruct is a large language model released by NVIDIA in October 2024. It is a customised version of Meta's Llama-3.1-70B-Instruct, adjusted so that its answers to user queries are more helpful.^[1] The model card states the aim plainly: improving the helpfulness of responses to user queries, with HelpSteer2 as its dataset and Llama-3.1-70B-Instruct as its base model.^[1]

How it was trained

The base model was tuned with reinforcement learning from human feedback, or RLHF. In RLHF a model is not trained against fixed target text but against a reward: candidate answers are scored, and the model's parameters are shifted so that higher-scoring answers become more likely. The score usually comes from a separate reward model, which has itself been trained on human preferences about which of two answers is better.

NVIDIA's recipe used the REINFORCE algorithm, the reward model Llama-3.1-Nemotron-70B-Reward, and prompts from HelpSteer2-Preference.^[1] REINFORCE is a policy-gradient method: it estimates how good a sampled answer turned out to be relative to expectation, then moves the model in the direction that makes good answers more probable. Because it needs only a score and not a single correct answer, it fits training on a reward signal.

Results

NVIDIA reported three automatic alignment benchmarks, all judged by GPT-4-Turbo: 85.0 on Arena Hard, 57.6 on AlpacaEval 2 LC, and 8.98 on MT-Bench.^[1] As of 1 October 2024 the model ranked first on all three, ahead of GPT-4o and Claude 3.5 Sonnet.^[1] Benchmarks of this kind ask a strong model to compare or score answers rather than collecting human votes; the LC figure for AlpacaEval 2 is length-controlled, meaning the win rate is adjusted to weaken the advantage of simply writing longer answers.

The human-voted Chatbot Arena gave a different picture. On 24 October 2024 the model had an Elo of 1267, ranking ninth, and 26th once style control was applied.^[1] Elo is a rating computed from pairwise wins and losses, and style control re-runs the ranking with corrections for the style of responses, so that length and formatting do not carry the score on their own. NVIDIA noted that the three automatic alignment benchmarks are known to be predictive of Chatbot Arena Elo.^[1] Put side by side, the two sets of numbers show how much a ranking depends on who is judging and what is being corrected for.

Practical details

The model accepts up to 128K input tokens and produces up to 4K output tokens.^[1] It is released under the NVIDIA Open Model License, with the Llama 3.1 Community License also applying.^[1] The -HF variant in the repository was converted from Llama-3.1-Nemotron-70B-Instruct so that it runs in the Transformers codebase, and NVIDIA warned that evaluation results might differ slightly.^[1]

Where it sits

The model follows a common pattern in open-weight AI: take a widely used base model — here Meta's Llama 3.1 family — and do further post-training aimed at one specific behaviour, in this case helpfulness. The name records both layers at once: Llama-3.1 for the base model, and Nemotron, which also appears in the name of the reward model used in training.^[1] Because the release carries NVIDIA's own licence and the Llama 3.1 Community License at the same time, anyone using it is subject to both.^[1]