Embodied AI

What it is

Embodied AI describes AI agents that learn through interaction with an environment rather than only from datasets of images, video or text curated from the internet.^[1] The distinction is about where the learning signal comes from. A system trained on internet material works over things that already exist; an embodied system has to act, observe what follows, and adjust. In the framing used in the field, the agent learns from interactions using an egocentric perception similar to that of humans: it sees the world from its own position rather than as a detached observer of a ready-made dataset.^[1]

That does not make datasets useless. It means that what an agent picks up from acting is different in kind from what it can pick up from a fixed collection of examples, because the agent's own choices shape what it encounters next.

Simulators as virtual testbeds

Teaching an agent by letting it act in physical space is slow, and mistakes can be expensive. The survey therefore frames simulators as virtual testbeds that let researchers train and test embodied frameworks before deploying them in the real world.^[2] Simulation also addresses a data problem: simulated environments make task-based dataset collection practical, because replicating the same settings in the real world requires extensive manual labour.^[2] A task defined once in a simulator can be run again and again under the same conditions, which is how a collection of task data can be built at all.

To compare the tools available, the survey evaluates nine state-of-the-art embodied AI simulators against seven features that it proposes, so that the simulators can be compared on how well each supports embodied research.^[1] The features are the survey's own yardstick, and the comparison is organised around them rather than around a single ranking.

Three research tasks

The survey groups work in the field into three main tasks: visual exploration, visual navigation and embodied question answering, and for each it covers state-of-the-art approaches, evaluation metrics and datasets.^[1] The names describe what the agent is asked to do with what it perceives — move through and inspect a scene, reach a place, or answer a question whose answer has to be found in the environment. What the three have in common is that performance depends on a sequence of actions and observations, not on a single pass over a fixed input. Each task comes with its own metrics and datasets, so results are reported task by task rather than in one common currency.

Where the framing came from

Duan and colleagues published the survey in March 2021, presenting it as a description of an emerging paradigm shift from what they call internet AI to embodied AI.^[1] The authors argue that growing interest in embodied AI is beneficial to the broader pursuit of artificial general intelligence.^[1] On that view, agents that have to cope with an environment are a step toward general-purpose systems rather than a separate niche alongside them.

What to keep in mind

The comparison the survey offers is bounded by its own choices: nine simulators measured against seven features.^[1] And because simulators are presented as a stage that comes before real-world deployment, how well results carry over from a simulated testbed to physical settings is central to how much any simulated result tells you.^[2] Read this way, the survey is a map of a fast-moving area as it stood at publication, together with the vocabulary — internet AI versus embodied AI, simulators as testbeds, tasks and metrics — that later work in the area could be described with.