Home›Concepts
Concepts
Concepts
How AI works, in plain language: from the artificial neuron to the Transformer, pre-training and reasoning models.
Total 46 · Clear filters
Artificial neural networksAn artificial neural network is a computing system of many simple connected units whose behaviour is set by connection weights. Its roots lie in 1943 models of neurons, and its learning versions began with Frank Rosenblatt's perceptron in 1958.Information theoryA framework that treats information as a measurable quantity to be encoded and transmitted, founded in Shannon's 1948 paper; it gave the field the bit, entropy and channel capacity.k-means clusteringk-means clustering splits n observations into k groups by repeatedly sending each point to its nearest centroid and recomputing each centroid as its cluster's mean. It is simple and scales to large data, but k must be fixed in advance.Support vector machineA supervised maximum-margin model for classification and regression that separates classes by the widest possible margin and uses kernel functions to handle non-linear data.Cross-validationA model validation technique that repeatedly splits data into folds, trains on some parts and validates on others, then averages the results to estimate how well a model or statistical result will generalise to unseen data.Convolutional neural networksA convolutional neural network applies the same small pattern detectors across a whole image, so a feature is found wherever it appears. Trained with backpropagation in 1989 and refined as LeNet-5 in 1998, it became standard in computer vision.TokenizationTokenization splits text into the tokens a language model actually reads; most modern systems do it with byte pair encoding, a 1994 compression method adapted for translation in 2015.AdaBoostAdaBoost, short for Adaptive Boosting, is a statistical classification meta-algorithm introduced by Yoav Freund and Robert Schapire in 1995. It reweights training examples, trains weak classifiers in sequence and combines their outputs into one weighted classifier.Topic modelTopic models represent documents as mixtures of latent topics, giving a compact statistical description of text collections. The 2003 Latent Dirichlet Allocation paper formalized this idea as a hierarchical Bayesian model.Deep learning on GPUsGraphics processing units (GPUs) run many arithmetic operations in parallel, which suits neural-network training. Their use from 2010 onward made deep learning practical at much larger scales.Attention mechanismAttention scores how important each element of a sequence is relative to the others and turns those scores into weights, letting a model read any part of its input directly. It is the operation at the core of the transformer.Attention and self-attentionAttention lets a neural network weigh the parts of its input by relevance. Self-attention applies this idea inside a single sequence, so each token gathers information from every other token. It is the core operation of the Transformer.Positional encodingA component of sequence models, especially transformers, that encodes each token's position as a vector, giving order-blind attention a sense of sequence; its forms range from the original transformer's sinusoids to RoPE and ALiBi.Mixture of experts (MoE)A mixture of experts model contains many sub-networks but routes each token to only a few of them. This lets total model size grow while the compute spent on each token stays much smaller than the total.Tensor CoresTensor Cores are specialised units inside NVIDIA GPUs that perform a small matrix multiply-and-add in one step, making the matrix arithmetic behind neural-network training and inference dramatically faster.TransformerThe Transformer is a neural network architecture introduced in 2017 that relies on self-attention instead of recurrence or convolution. Most of today's large language models are built on it.Model parallelismModel parallelism splits one model's parameters and computation across multiple accelerators, so networks too large for a single device can still be trained. NVIDIA's 2019 Megatron-LM paper showed an intra-layer version built in native PyTorch.Retrieval-augmented generationRetrieval-augmented generation (RAG) lets a language model fetch relevant documents from an external store and put them into its prompt before answering, so it can use knowledge beyond its training data and cite where an answer came from.Scaling lawsScaling laws are empirical trends showing that a language model's test loss falls smoothly, as a power law, when parameters, data and training compute grow. They help plan large training runs and explain the race for compute.Content provenance (C2PA)Content provenance is a checkable record of where a piece of media came from and how it was edited, attached to the file itself. The C2PA, founded in 2021 by Adobe, Arm, BBC, Intel, Microsoft and Truepic, publishes the technical standard behind it.