Home›Concepts
Concepts
Concepts
How AI works, in plain language: from the artificial neuron to the Transformer, pre-training and reasoning models.
Total 19 · Clear filters
Boltzmann machineA Boltzmann machine is a neural network whose configurations are scored by a global energy and trained by a stochastic learning algorithm. Introduced by Geoffrey Hinton and Terry Sejnowski, it brought energy-based models into cognitive science and machine learning.Topic modelTopic models represent documents as mixtures of latent topics, giving a compact statistical description of text collections. The 2003 Latent Dirichlet Allocation paper formalized this idea as a hierarchical Bayesian model.Deep neural network acoustic modelingDeep neural network acoustic modeling uses multilayer networks to map speech to units such as phones. A 2010 NIPS paper used Deep Belief Nets and a mean-covariance RBM to extract speech features, reaching 20.5% phone error on TIMIT.Deep learning on GPUsGraphics processing units (GPUs) run many arithmetic operations in parallel, which suits neural-network training. Their use from 2010 onward made deep learning practical at much larger scales.Variational autoencoderA variational autoencoder is a generative model whose encoder maps an input to a distribution over a latent space and whose decoder reconstructs the input from a sample; training maximises a lower bound on the data likelihood.Generative adversarial networkA generative adversarial network pits a generator that makes fake data against a discriminator that judges it, both trained in a zero-sum game; proposed by Ian Goodfellow and colleagues in 2014, it led to a series of architectures for generating images.DeepfakeDeepfake is synthetic media: images, video or audio made or altered with AI. The term, a blend of "deep learning" and "fake", appeared in late 2017 and now names a whole class of face-swapping and manipulation techniques.Mixture of experts (MoE)A mixture of experts model contains many sub-networks but routes each token to only a few of them. This lets total model size grow while the compute spent on each token stays much smaller than the total.Tensor CoresTensor Cores are specialised units inside NVIDIA GPUs that perform a small matrix multiply-and-add in one step, making the matrix arithmetic behind neural-network training and inference dramatically faster.Model parallelismModel parallelism splits one model's parameters and computation across multiple accelerators, so networks too large for a single device can still be trained. NVIDIA's 2019 Megatron-LM paper showed an intra-layer version built in native PyTorch.Scaling lawsScaling laws are empirical trends showing that a language model's test loss falls smoothly, as a power law, when parameters, data and training compute grow. They help plan large training runs and explain the race for compute.Diffusion modelsDiffusion models learn to reverse a gradual noising process, so they can turn random noise into a new image by removing noise step by step. Latent diffusion, which runs this in a compressed space, became the basis of Stable Diffusion.Speculative decodingSpeculative decoding speeds up text generation from large autoregressive models: a smaller model proposes several tokens and the large model checks them in parallel, producing identical output faster.Text watermarkingText watermarking embeds a hidden statistical signal in generated text so that a short span of tokens can later be identified as machine output, while the text still reads normally to people.Sim-to-real transferSim-to-real transfer is the practice of training robot policies in simulation and then adapting them to physical robots, using synthetic data to supplement costly real-world demonstrations.Unsupervised learningUnsupervised learning trains algorithms on data that carry no labels, letting them find structure such as clusters, low-dimensional representations or generative models. It underlies clustering tools and neural generative models from the Hopfield network and Boltzmann machine to the variational autoencoder.QuantizationQuantization stores a language model's weights and activations at lower numerical precision, cutting the memory and bandwidth that inference requires. Methods such as LLM.int8() and GPTQ made very large models runnable on far smaller hardware.KV cacheA KV cache keeps the key and value vectors a model has already computed for earlier tokens, so each new token reuses them instead of recomputing. Its memory footprint is what makes high-throughput serving hard to scale.Synthetic dataSynthetic data is training material written by a model rather than collected from people or the web. In the phi-1 experiments a small code model trained partly on model-generated "textbooks" reached competitive benchmark scores, suggesting data quality itself shapes capability.