Home›Concepts
Concepts
Concepts
How AI works, in plain language: from the artificial neuron to the Transformer, pre-training and reasoning models.
Total 93
Deep neural network acoustic modelingDeep neural network acoustic modeling uses multilayer networks to map speech to units such as phones. A 2010 NIPS paper used Deep Belief Nets and a mean-covariance RBM to extract speech features, reaching 20.5% phone error on TIMIT.Deep learning on GPUsGraphics processing units (GPUs) run many arithmetic operations in parallel, which suits neural-network training. Their use from 2010 onward made deep learning practical at much larger scales.Xavier (Glorot) initializationXavier (Glorot) initialization is a weight-initialization scheme from Glorot and Bengio's 2010 paper that sets each layer's starting weight scale so signals stay stable across layers, giving substantially faster convergence in deep feedforward networks.DropoutA regularisation technique for neural networks: during training it randomly zeroes the inputs or outputs of some neurons so that units cannot co-adapt, which reduces overfitting.Variational autoencoderA variational autoencoder is a generative model whose encoder maps an input to a distribution over a latent space and whose decoder reconstructs the input from a sample; training maximises a lower bound on the data likelihood.Generative adversarial networkA generative adversarial network pits a generator that makes fake data against a discriminator that judges it, both trained in a zero-sum game; proposed by Ian Goodfellow and colleagues in 2014, it led to a series of architectures for generating images.Attention mechanismAttention scores how important each element of a sequence is relative to the others and turns those scores into weights, letting a model read any part of its input directly. It is the operation at the core of the transformer.Attention and self-attentionAttention lets a neural network weigh the parts of its input by relevance. Self-attention applies this idea inside a single sequence, so each token gathers information from every other token. It is the core operation of the Transformer.Batch normalizationA technique that re-centres each layer's inputs to zero and rescales them to unit variance over a mini-batch, making deep networks faster and more stable to train.Residual connectionA residual connection adds a block's input to its output, so the block learns only the change it needs to make. This makes very deep networks trainable, and it is a standard part of today's transformers.Knowledge distillationKnowledge distillation trains a small model to reproduce the outputs of a larger model or an ensemble, so that much of the predictive performance is kept at lower compute and deployment cost.Positional encodingA component of sequence models, especially transformers, that encodes each token's position as a vector, giving order-blind attention a sense of sequence; its forms range from the original transformer's sinusoids to RoPE and ALiBi.DeepfakeDeepfake is synthetic media: images, video or audio made or altered with AI. The term, a blend of "deep learning" and "fake", appeared in late 2017 and now names a whole class of face-swapping and manipulation techniques.Mixture of experts (MoE)A mixture of experts model contains many sub-networks but routes each token to only a few of them. This lets total model size grow while the compute spent on each token stays much smaller than the total.Tensor CoresTensor Cores are specialised units inside NVIDIA GPUs that perform a small matrix multiply-and-add in one step, making the matrix arithmetic behind neural-network training and inference dramatically faster.TransformerThe Transformer is a neural network architecture introduced in 2017 that relies on self-attention instead of recurrence or convolution. Most of today's large language models are built on it.Reinforcement learning from human feedback (RLHF)Reinforcement learning from human feedback trains a model toward what people prefer, using a reward model learned from human comparisons of its outputs. It became a standard step in building chat assistants.Pre-training and fine-tuningPre-training trains a large model on huge amounts of unlabeled text, and fine-tuning then adapts it to a specific task with a small labeled dataset. This two-stage recipe is the basis of the term foundation model.Model parallelismModel parallelism splits one model's parameters and computation across multiple accelerators, so networks too large for a single device can still be trained. NVIDIA's 2019 Megatron-LM paper showed an intra-layer version built in native PyTorch.Retrieval-augmented generationRetrieval-augmented generation (RAG) lets a language model fetch relevant documents from an external store and put them into its prompt before answering, so it can use knowledge beyond its training data and cite where an answer came from.