Fine-tuning large language models: a practical start

Who this is for and what you will get

This guide is for developers who have prototyped with prompt engineering or retrieval augmented generation (RAG) and still find the results inconsistent. By the end you will know whether your use case justifies fine-tuning, what a fine-tuning job needs, and how to choose a training technique and base model on Microsoft Foundry, prepare the data and evaluate the result.^[1]

What fine-tuning is and how it differs from prompting and RAG

Prompt engineering and RAG both work on the input: you change the prompt, or add retrieved data to it, to improve the answer. Fine-tuning works differently. It retrains the model itself with additional data.^[1]

Large language models are pre-trained on large amounts of text. Fine-tuning takes a pre-trained model and retrains it on a curated set of examples for a task or domain, which can produce a custom model that is more accurate and relevant there.^[1]

Few-shot prompting puts a few examples in the prompt, and it has two limits. Model token limits cap how many examples fit, and adding examples to every prompt makes token costs high. A side benefit of fine-tuning is that it can reduce the number of examples you need in the prompt, which cuts token usage and cost.^[1]

The fine-tuning discussed here is supervised: the retraining adds new data that was not part of the model's earlier training data.^[1]

When fine-tuning is worth it

Fine-tuning is an advanced technique that needs real expertise. Done incorrectly, it may not improve results and may even make the model worse in your target domain.^[1] Before you start, answer a few questions:^[1]

1. Use case: what aspect of the pre-trained model do you want to improve?^[1]
2. Alternatives: have you tried prompt engineering and RAG? Use them to build a baseline for comparison.^[1]
3. Costs: can the model be fine-tuned at all; how much effort goes into preparing data, evaluating and refining it; how much compute the training and deployment need; and do you have enough quality examples?^[1]
4. Benefits: does the fine-tuned model beat the baseline; does it cut token usage by simplifying prompts; can the base model be reused for new domains?^[1]

Fine-tuning is worth doing only when the benefits outweigh the costs.^[1] Once you decide to go ahead, you need a pre-trained model, a dataset, a training environment for the job, and a hosting environment for the fine-tuned model.^[1]

Fine-tuning on Microsoft Foundry

Microsoft Foundry is where you fine-tune, deploy and manage custom models on Azure. It brings together what used to be Azure OpenAI Studio and Azure AI Studio.^[1] Foundry uses LoRA (low-rank adaptation) to fine-tune efficiently. Training is faster and cheaper than retraining every weight.^[1]

The steps below cover the choices in order.

1. Choose a training technique. Foundry supports Supervised Fine-Tuning (SFT), Direct Preference Optimization (DPO) and Reinforcement Fine-Tuning (RFT). Start with SFT, which covers the widest range of scenarios.^[1] - SFT trains on input and output example pairs. It is the default for most tasks, such as domain specialization, style and tone, and instruction-following.^[1] - DPO learns from pairs of preferred and non-preferred responses, which aligns outputs with human preferences. It suits cases where you have comparative feedback.^[1] - RFT uses reward signals from graders and reinforcement learning to optimize complex behavior. It suits math, chemistry and physics, where answers are clearly right or wrong, and it requires more machine learning expertise.^[1] 2. Choose a training tier. Standard trains in your resource's region and guarantees data residency, which suits data that must stay in one region.^[1] Global is cheaper and queues faster by using capacity beyond your region, and data and weights are copied to the training region.^[1] Developer has the lowest cost. It uses idle capacity with no latency or SLA guarantees, jobs can be preempted and resumed, and it suits experiments.^[1] 3. Choose a base model. Fine-tunable models include OpenAI's gpt-4o-mini, gpt-4o, the gpt-4.1 family, o4-mini and gpt-5, plus open-source models such as Ministral-3B, Qwen-32B, Llama-3.3-70B-Instruct and gpt-oss-20b. Supported methods, regions and availability differ, so check the official fine-tuning model list before you start.^[1] 4. Choose a modality. Serverless uses consumption-based pricing and needs no GPU quota to manage. It covers OpenAI and selected models.^[1] Managed compute lets you bring your own VMs through Azure Machine Learning, and it offers the widest model range.^[1] Most people should start with serverless.^[1]

Practical guidance for data and training

• Set a baseline first. Measure the base model with prompt engineering and RAG before you fine-tune, so you can show the gain.^[1]
• Start small, then scale. Begin with 50 to 100 high-quality examples to validate the approach, then grow to 500 or more for production. Quality beats quantity, so remove low-quality examples.^[1]
• Format the files correctly. Training and validation files must be JSONL, UTF-8 with a BOM, under 512 MB, and use the chat-completions message format. Always include a validation file so you can watch for overfitting.^[1]
• Keep the system message consistent. Use the same system message at inference that you used during training.^[1]
• Compare checkpoints. Foundry keeps the last three epochs as deployable checkpoints. Watch train_loss, valid_loss and token accuracy, and choose the checkpoint that generalizes best instead of deploying the last one.^[1]
• Measure cost with quality. When you compare with the baseline, measure token cost alongside output quality.^[1]
• Continue from a fine-tuned model. For OpenAI models you can fine-tune an already fine-tuned model on new data.^[1]
• Watch hosting costs. A deployed custom model bills hourly, and an inactive deployment is removed after 15 days, so clean up what you do not need.^[1]

Common questions

If the result is disappointing, the usual suspects are too little or low-quality data, or no baseline to compare against. This is an inference from the guidance above, not a troubleshooting guide from Microsoft.^[1]

Model lists, regions and usage numbers change as the platform is updated. Check the official Foundry documentation before you start.^[1]

Practical resources to try

OpenAI's cookbook has a worked example, How to fine-tune chat models. It fine-tunes a recent chat model for a recipe assistant, covering training data preparation, the job, and inference with the new model.^[1] The Microsoft Foundry walkthrough Customize a model with fine-tuning fine-tunes a model such as gpt-4.1-mini on Azure. It covers uploading training and validation data, running the job, then deploying and using the model.^[1] A Hugging Face post uses the transformers library and TRL (Transformer Reinforcement Learning) to fine-tune an open model such as CodeLlama 7B on open datasets.^[1]

If you want to write less code, look at AutoTrain. It is a Python library and no-code solution from Hugging Face, and it runs in your own cloud, on Hugging Face Spaces or locally, with a web GUI, a CLI and YAML configuration files.^[1] Unsloth is an open-source framework for LLM fine-tuning and reinforcement learning, with ready-to-use notebooks.^[1]

Next steps

Walking through one of these resources end to end will show you the cost and the benefit more clearly than reading about them. Then return to your own data and decide whether fine-tuning is really needed.^[1]

Adapted from lesson 18 of Microsoft's open-source course *Generative AI for Beginners*^[1], released under the MIT License.