PyTorch

What PyTorch is

PyTorch is an open-source framework for deep learning — the part of machine learning built on neural networks with many layers. A framework of this kind supplies the pieces a model needs: tensors (multi-dimensional arrays of numbers), the operations performed on them, and the machinery for computing gradients, the quantities that tell a training algorithm how to adjust the model's parameters. PyTorch's original authors are Adam Paszke, Sam Gross, Soumith Chintala and Gregory Chanan. The code is written in Python, C++ and CUDA (NVIDIA's programming model for running code on graphics processors), and it is released under the BSD-3 licence.^[2]

Where it came from

Development began in 2016, with people from the Lua Torch community working alongside a large share of people and funding from Meta, then still called Facebook.^[2] The result is a framework that is driven mainly from Python, the language many researchers already use for data work, while the parts that need speed are written in C++ and CUDA.

The first public beta appeared in January 2017; the release notes record a public release on 18 January.^[3]

From research tool to production

For its first years PyTorch was chiefly a research tool. In 2018 Facebook merged Caffe2, its own framework aimed at production deployment, into PyTorch, so that the project would be ready for production use as well as for experiments.^[4][2]

PyTorch 1.0 followed on 7 December 2018. It added a JIT compiler (torch.jit), which takes model code and compiles it into a form that can be run and optimised outside an interactive Python session — the usual way to move a trained model into a serving system. The release also brought a redesigned distributed package, for spreading training across many processors or machines; a C++ frontend, for programs that use trained models without going through Python; and Torch Hub, a place to publish and fetch pre-trained models.^[5]

The PyTorch Foundation

On 12 September 2022 Meta moved the project to the Linux Foundation, where it became the PyTorch Foundation, with a governing board made up of AMD, AWS, Google Cloud, Meta, Microsoft Azure and NVIDIA.^[6][1] The change put the framework under a neutral body in which several competing companies sit on the board, rather than under a single vendor. At the time of the move the project had more than 2,400 contributors, and Mark Zuckerberg said that more than 150,000 GitHub projects had been built on it.^[1][6]

PyTorch 2.0

PyTorch 2.0 was released on 15 March 2023 and introduced TorchDynamo, a compiler that works at the Python level.^[2] A compiler of this kind reads the model code a user has already written and turns it into a lower-level, faster form, rather than requiring the model to be rewritten in another language first.

Where it is used

PyTorch is one of the frameworks on which large models are actually trained and run. OpenAI standardised on PyTorch in January 2020, and Hugging Face's Transformers library and Tesla's Autopilot are also built on it.^[2]

Meta has used its on-device path, PyTorch ExecuTorch, for Llama 3.2 distributions aimed at edge devices, and has said that the PyTorch project supported Llama 3.1 from the day it was released.^[7][8] On-device here means the model runs on the phone or other local hardware rather than in a data centre, which is a different job from training: the constraints are memory and power rather than raw throughput.

The sequence — a research framework, then a production one, then a neutrally governed project, then a compiled one — is roughly the path a piece of research software takes when it becomes shared infrastructure that many organisations depend on.