Theano

What it is

Theano is a Python library for mathematical work with multi-dimensional arrays. Its documentation describes it as a library that lets users define, optimize and evaluate mathematical expressions involving multi-dimensional arrays efficiently^[1]. It is also described as a Python library and optimizing compiler for mathematical expressions, especially matrix-valued ones, compiled to run efficiently on CPU or GPU architectures^[2]. Both descriptions point at the same design: the user writes an expression in Python, and the library takes responsibility for turning it into something that runs fast, instead of executing it step by step the way ordinary interpreted code would. An optimizing compiler, in the ordinary sense, takes a description of a computation and rewrites it into an equivalent one that runs faster; Theano does this for array arithmetic.

Because it was first released in 2007, Theano was primarily developed by the Montreal Institute for Learning Algorithms at the Université de Montreal^[2]. Its documentation states that it had been powering large-scale computationally intensive scientific investigations since 2007^[1] — numerically heavy work in which the cost of the arithmetic dominates everything else. That is the setting the library was built for.

Main features

The documentation lists tight integration with NumPy, transparent use of a GPU, efficient symbolic differentiation, speed and stability optimizations, and dynamic C code generation^[1]. Each item addresses a different practical obstacle.

Integration with NumPy means that arrays built and manipulated with NumPy can be used inside Theano expressions without extra conversion steps, so existing code and working habits carry over.

Transparent use of a GPU means the same program can be sent to a graphics processor; the documentation says GPU use can perform data-intensive calculations up to 140 times faster than with a CPU, for float32 only^[1]. The qualifier matters: that acceleration applies to 32-bit floating-point arithmetic, so work that needs higher precision does not benefit in the same way.

Symbolic differentiation follows from the way the library stores an expression: not as a sequence of operations already carried out, but as a description of a computation. Because the description is available, the library can work out the derivatives of the expression for the user. Many numerical optimization procedures spend their preparatory effort deriving gradients by hand; this step is handled automatically instead.

Speed and stability optimizations, together with dynamic C code generation, mean the expression is rewritten into a better form and turned into compiled machine code at run time rather than interpreted.

In the classroom and in the literature

The documentation says Theano was approachable enough for classroom use, citing the University of Montreal course IFT6266^[1]. Students could write array computations in ordinary Python and leave compilation and device handling to the library, rather than learning a separate compiled language for a course project.

For published work done with Theano, the documentation asks users to cite two papers: Theano: A CPU and GPU Math Expression Compiler, presented at SciPy 2010 and written by Bergstra, Breuleux, Bastien, Lamblin, Pascanu, Desjardins, Turian, Warde-Farley and Bengio^[1]; and Theano: new features and speed improvements, a NIPS 2012 deep learning workshop paper^[1]. The request shows how the project wanted to be credited — through the two publications that documented its performance and its newer capabilities.

End of major development

On 28 September 2017, Pascal Lamblin posted a message from Yoshua Bengio saying that major development would cease after the 1.0 release^[2].

What it did

Put plainly: Theano let a researcher describe array computations in Python, differentiate them automatically, and run them on a GPU without rewriting the program for that device. It was used for large-scale scientific computation from 2007, and it was simple enough to be taught in a university course^[1]. Major development ended in 2017^[2]; the software and its documentation remain as a record of one way of pairing a high-level language with a compiling backend for numerical work.