Home›Concepts
Concepts
Concepts
How AI works, in plain language: from the artificial neuron to the Transformer, pre-training and reasoning models.
Total 48 · Clear filters
Boltzmann machineA Boltzmann machine is a neural network whose configurations are scored by a global energy and trained by a stochastic learning algorithm. Introduced by Geoffrey Hinton and Terry Sejnowski, it brought energy-based models into cognitive science and machine learning.Backpropagation and gradient descentBackpropagation is the procedure that lets a multilayer neural network learn by sending output errors backward to adjust its weights, using gradient descent. A 1986 Nature paper made it widely known, and it is standard in most neural networks today.TokenizationTokenization splits text into the tokens a language model actually reads; most modern systems do it with byte pair encoding, a 1994 compression method adapted for translation in 2015.Topic modelTopic models represent documents as mixtures of latent topics, giving a compact statistical description of text collections. The 2003 Latent Dirichlet Allocation paper formalized this idea as a hierarchical Bayesian model.Connectionist Temporal ClassificationConnectionist temporal classification (CTC) is a method for training recurrent neural networks to label unsegmented sequences, such as speech or handwriting, without frame-by-frame alignment. Introduced in 2006, it let CTC-trained LSTMs win the first international contests for recurrent networks.Deep neural network acoustic modelingDeep neural network acoustic modeling uses multilayer networks to map speech to units such as phones. A 2010 NIPS paper used Deep Belief Nets and a mean-covariance RBM to extract speech features, reaching 20.5% phone error on TIMIT.Xavier (Glorot) initializationXavier (Glorot) initialization is a weight-initialization scheme from Glorot and Bengio's 2010 paper that sets each layer's starting weight scale so signals stay stable across layers, giving substantially faster convergence in deep feedforward networks.DropoutA regularisation technique for neural networks: during training it randomly zeroes the inputs or outputs of some neurons so that units cannot co-adapt, which reduces overfitting.Variational autoencoderA variational autoencoder is a generative model whose encoder maps an input to a distribution over a latent space and whose decoder reconstructs the input from a sample; training maximises a lower bound on the data likelihood.Generative adversarial networkA generative adversarial network pits a generator that makes fake data against a discriminator that judges it, both trained in a zero-sum game; proposed by Ian Goodfellow and colleagues in 2014, it led to a series of architectures for generating images.Batch normalizationA technique that re-centres each layer's inputs to zero and rescales them to unit variance over a mini-batch, making deep networks faster and more stable to train.Residual connectionA residual connection adds a block's input to its output, so the block learns only the change it needs to make. This makes very deep networks trainable, and it is a standard part of today's transformers.Knowledge distillationKnowledge distillation trains a small model to reproduce the outputs of a larger model or an ensemble, so that much of the predictive performance is kept at lower compute and deployment cost.Positional encodingA component of sequence models, especially transformers, that encodes each token's position as a vector, giving order-blind attention a sense of sequence; its forms range from the original transformer's sinusoids to RoPE and ALiBi.DeepfakeDeepfake is synthetic media: images, video or audio made or altered with AI. The term, a blend of "deep learning" and "fake", appeared in late 2017 and now names a whole class of face-swapping and manipulation techniques.TransformerThe Transformer is a neural network architecture introduced in 2017 that relies on self-attention instead of recurrence or convolution. Most of today's large language models are built on it.Pre-training and fine-tuningPre-training trains a large model on huge amounts of unlabeled text, and fine-tuning then adapts it to a specific task with a small labeled dataset. This two-stage recipe is the basis of the term foundation model.Model parallelismModel parallelism splits one model's parameters and computation across multiple accelerators, so networks too large for a single device can still be trained. NVIDIA's 2019 Megatron-LM paper showed an intra-layer version built in native PyTorch.Scaling lawsScaling laws are empirical trends showing that a language model's test loss falls smoothly, as a power law, when parameters, data and training compute grow. They help plan large training runs and explain the race for compute.In-context learningIn-context learning is a large language model's ability to perform a new task from instructions and a few examples placed in the prompt, with no weight updates. Named and studied in the GPT-3 paper, it moved adaptation from fine-tuning to prompt writing.