Cloud Concepts
| Test-time compute Test-time compute is the computation a model spends while answering a prompt, beyond what training already fixed into its weights. A 2024 study by Snell et al. showed how to spend it adaptively, and how it can stand in for a bigger model. | |
| KV cache A KV cache keeps the key and value vectors a model has already computed for earlier tokens, so each new token reuses them instead of recomputing. Its memory footprint is what makes high-throughput serving hard to scale. |