UI-TARS

What it is

UI-TARS is a graphical user interface (GUI) agent model developed at ByteDance Seed. Its input is a screenshot; its output is a keyboard or mouse action^[1]. Nothing else is handed to it: the model sees the screen as a picture and decides where to click, what to type and when to scroll.

The word "native" in the name points at this design. Instead of pairing a general-purpose language model with an external script that parses its text and turns it into clicks, UI-TARS keeps perception, reasoning and action inside one model trained for the job. The loop between seeing the screen and acting on it is therefore short, and the same model can be aimed at different applications rather than at one fixed workflow.

Where it came from

The work is described in a paper submitted to arXiv on 21 January 2025 by Yujia Qin and 34 co-authors^[1].

How it works

The paper's abstract names four innovations^[1]. The first is perception: the model's ability to read a screen was strengthened with a large dataset of GUI screenshots, so that it can recognise buttons, text fields, menus and other elements across many kinds of software. The second is unified action modelling across platforms — one set of keyboard and mouse actions serves the agent whichever system it is working on, rather than a separate action vocabulary being designed for each platform.

The third is System-2 style reasoning. In cognitive terms, "System 2" describes slow, deliberate thinking as opposed to fast reflex. An agent that clicks immediately after seeing a single screenshot can carry one mistake forward into the next step; an agent that reasons about the current state of the screen before choosing an action has a better chance of recovering. The fourth innovation is iterative training on reflective online interaction traces: the model is trained on records of its own interactions, including reflection on what went wrong.

Results

The paper reports state-of-the-art results on more than 10 GUI-agent benchmarks, covering perception, grounding — locating the element on screen that should be operated — and task execution^[1]. On OSWorld it scored 24.6 when allowed 50 steps and 22.7 with 15 steps, against 22.0 and 14.9 for Claude^[1]. On AndroidWorld it scored 46.6 against 34.5 for GPT-4o^[1]. The step counts matter: on both sides, widening the budget from 15 to 50 steps raised the score, which shows how strongly this kind of task depends on how many attempts the agent is allowed.

Release

The model was released with open weights under Apache-2.0 on GitHub, including the UI-TARS-72B-DPO model^[2]. The project also ships UI-TARS-desktop, for running the agent on a personal computer^[2]. The repository updated its OSWorld inference scripts on 23 March 2025^[2].

What came after

UI-TARS was succeeded by UI-TARS-1.5 on 16 April 2025, announced in a progress blog post, when UI-TARS-1.5-7B was open-sourced^[2]. UI-TARS-2 followed on 4 September 2025^[2].

Why it matters

Most software is built for human eyes and hands: a person has to find the button and fill in the form. A GUI agent tries to let a model take over those interfaces directly, without every application first having to offer a programmatic interface for machines. Doing that from screenshots alone is the hard part, because the model has to work out what it is looking at and what a reasonable next action is, at every single step, without a map of the application. UI-TARS is one attempt at building that capability into a single model, and its open weights made the approach available for others to run and compare against.