Adept releases the encoder-free multimodal model Fuyu-8B

Background

Adept framed Fuyu-8B as one component of a general copilot for knowledge workers, software meant to carry out work on a person's behalf rather than only answer questions in a chat window^[1]. An assistant of that kind has to read a screen the way a person does, which is why the visual half of the model carried as much weight in the announcement as the text half.

The post also made an argument about how multimodal models should be built. Adept wrote that existing multimodal models were too complicated, both architecturally and in training, to debug, scale and deploy^[1]. That framing presented the release as a simplification of the prevailing recipe rather than a new record on any benchmark.

Release

Adept published Fuyu-8B on 17 October 2023^[1]. The model is described as an encoder-free multimodal architecture. In the common arrangement, an image first passes through a dedicated vision encoder that converts it into features, and a language model then works on those features; calling a model encoder-free means it does not follow that two-stage split, and the picture reaches the model without a separate vision encoder standing in between.

Weights and example code went up on Hugging Face the same day^[1]. Publishing weights alongside the announcement changes who can examine a model: outside developers can download and run it, not merely read a description of it. It also lets other teams check the architectural claim for themselves instead of taking the post's word for it.

What was held back

The release was bounded in ways the post stated plainly. Adept said the larger Fuyu-Medium would not be released^[1]. It also previewed screen localisation, meaning the ability to locate a specific element on a screen, and question answering about a user interface, but said those capabilities were reserved for internal models^[1]. And it said a product alpha would follow^[1].

Read together, those three points describe a staged rollout: a smaller model out in the open now, the larger model and the interface-level skills kept in-house, and a product still to come. The split also marks where the company drew the line between what counts as shared groundwork and what it treats as its own commercial territory.

Why it mattered

For anyone building agents, a multimodal model with downloadable weights is something to build on directly, without going through a closed interface. Fuyu-8B's contribution was less a capability claim than a design position: that the visual side of a multimodal model can be simplified, and that a simpler model is easier to debug, scale and deploy^[1]. Because the weights and example code were public from day one^[1], that position could be tested by people outside Adept rather than argued about in the abstract.

The emphasis on screens rather than on photographs is equally telling. Screen localisation and interface question answering are the skills an assistant needs before it can click the right button or fill in the right field^[1], and Adept chose to keep those in reserve while releasing the base model. The announcement therefore reads as a snapshot of an unfinished sequence: a model out, a product alpha promised^[1], and the hardest part of the agent problem still behind the door.