Code Papers
| Evaluating Large Language Models Trained on Code (Codex / HumanEval) A 2021 paper from OpenAI introduces Codex, a GPT model fine-tuned on public GitHub code, and HumanEval, an evaluation set that judges generated Python programs by whether they work rather than by how closely they match a reference. | |
| SWE-bench: Can Language Models Resolve Real-World GitHub Issues? A benchmark of 2,294 real software engineering problems taken from GitHub issues and the pull requests that resolved them, in which a model must edit a codebase to fix an issue; on release the best model solved under 2% of them. | |
| GLM-5: from Vibe Coding to Agentic Engineering Zhipu's GLM-5 report describes a 744-billion-parameter mixture-of-experts model built on sparse attention and asynchronous reinforcement learning, aimed at moving AI-assisted programming from vibe coding toward agentic engineering. | |
| Program Synthesis with Large Language Models (MBPP) A 2021 arXiv paper by Austin, Odena, Nye and eight other authors that introduced MBPP, a benchmark of 974 short Python programming tasks, and measured how language models from 244M to 137B parameters turn natural-language descriptions into working code. | |
| LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code A benchmark for code LLMs that keeps collecting fresh contest problems from LeetCode, AtCoder and CodeForces, so that scores reflect ability rather than memorised test data, and that also tests code repair and execution reasoning. | |
| Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfaces Terminal-Bench 2.0 is a curated benchmark of 89 hard tasks carried out in computer terminal environments, built to measure how well AI agents handle realistic command-line work; frontier models and agents score below 65% on it. |