Home›Concepts
Concepts
Concepts
How AI works, in plain language: from the artificial neuron to the Transformer, pre-training and reasoning models.
Total 63 · Clear filters
Variational autoencoderA variational autoencoder is a generative model whose encoder maps an input to a distribution over a latent space and whose decoder reconstructs the input from a sample; training maximises a lower bound on the data likelihood.Generative adversarial networkA generative adversarial network pits a generator that makes fake data against a discriminator that judges it, both trained in a zero-sum game; proposed by Ian Goodfellow and colleagues in 2014, it led to a series of architectures for generating images.Attention mechanismAttention scores how important each element of a sequence is relative to the others and turns those scores into weights, letting a model read any part of its input directly. It is the operation at the core of the transformer.Attention and self-attentionAttention lets a neural network weigh the parts of its input by relevance. Self-attention applies this idea inside a single sequence, so each token gathers information from every other token. It is the core operation of the Transformer.Batch normalizationA technique that re-centres each layer's inputs to zero and rescales them to unit variance over a mini-batch, making deep networks faster and more stable to train.Residual connectionA residual connection adds a block's input to its output, so the block learns only the change it needs to make. This makes very deep networks trainable, and it is a standard part of today's transformers.Knowledge distillationKnowledge distillation trains a small model to reproduce the outputs of a larger model or an ensemble, so that much of the predictive performance is kept at lower compute and deployment cost.Positional encodingA component of sequence models, especially transformers, that encodes each token's position as a vector, giving order-blind attention a sense of sequence; its forms range from the original transformer's sinusoids to RoPE and ALiBi.Tensor CoresTensor Cores are specialised units inside NVIDIA GPUs that perform a small matrix multiply-and-add in one step, making the matrix arithmetic behind neural-network training and inference dramatically faster.TransformerThe Transformer is a neural network architecture introduced in 2017 that relies on self-attention instead of recurrence or convolution. Most of today's large language models are built on it.Pre-training and fine-tuningPre-training trains a large model on huge amounts of unlabeled text, and fine-tuning then adapts it to a specific task with a small labeled dataset. This two-stage recipe is the basis of the term foundation model.Diffusion modelsDiffusion models learn to reverse a gradual noising process, so they can turn random noise into a new image by removing noise step by step. Latent diffusion, which runs this in a compressed space, became the basis of Stable Diffusion.State space model (Mamba)A state space model represents a sequence as a hidden state that evolves with the input. S4 brought the idea into deep learning in 2021, and Mamba made its parameters input-dependent, enabling linear-time modeling of long sequences.Content provenance (C2PA)Content provenance is a checkable record of where a piece of media came from and how it was edited, attached to the file itself. The C2PA, founded in 2021 by Adobe, Arm, BBC, Intel, Microsoft and Truepic, publishes the technical standard behind it.Speculative decodingSpeculative decoding speeds up text generation from large autoregressive models: a smaller model proposes several tokens and the large model checks them in parallel, producing identical output faster.Text watermarkingText watermarking embeds a hidden statistical signal in generated text so that a short span of tokens can later be identified as machine output, while the text still reads normally to people.Prompt injectionPrompt injection is an attack in which text inside a model's input overrides its original instructions, letting an adversary hijack a language-model application — including, as researchers showed in 2023, through content the application retrieves on its own.Open Source AI Definition (OSAID)The Open Source AI Definition is a 1.0 standard, released by the Open Source Initiative in October 2024, that a model must meet to be called open source: reproducible design, disclosed training data, and freedom to use, modify and build on it.Vision-Language-Action model (VLA)A vision-language-action model (VLA) is a robot policy that takes in camera images and a language instruction and outputs actions directly, usually by adapting a pretrained vision-language model to control.Sim-to-real transferSim-to-real transfer is the practice of training robot policies in simulation and then adapting them to physical robots, using synthetic data to supplement costly real-world demonstrations.