Fuyu-8B

What it is

Fuyu-8B is a multimodal transformer that Adept released as open source on 17 October 2023^[1]. Its distinguishing feature is architectural: it is a decoder-only transformer with no image encoder at all, and image patches are projected directly into the first layer of the network^[1][2]. In most vision-language models a picture first passes through a separate vision backbone, and the resulting representation is handed to a language model. Fuyu-8B has only one stack, and the image enters it at the bottom, as tokens would.

Why the missing encoder matters

Because the model drops position embeddings that are specific to images, it accepts arbitrary image resolutions, both during training and at inference^[1]. Resolution is usually something a vision encoder fixes in advance, so that pictures must be resized or tiled to fit; here that constraint is removed, and a tall screenshot or a wide slide can go in as it is.

Adept also says the model answers questions about large images in under 100 milliseconds^[1]. That figure matters because of where the model is meant to sit: inside an interactive loop, where a delay of a second or more per step would make the whole thing unusable.

What the numbers say

Adept reports scores of 74.2 on VQAv2, 60.6 on OKVQA and 64.5 on AI2D^[1][2]. The company states that Fuyu-8B beats Qwen-VL and PaLM-E-12B on two of those three metrics while using 2 billion and 4 billion fewer parameters respectively^[1]. On COCO captioning it reaches a score of 141, above the 138 of Fuyu-Medium, a larger model that Adept had not released^[1].

These are benchmark scores on standard visual question answering and diagram datasets, useful for placing the model against contemporaries but not a description of how it behaves in open-ended use.

What it was built for

Fuyu-8B was designed to read screens, charts, slides and PDFs — the inputs a copilot needs when it has to click and scroll through interfaces on the user's behalf^[1]. Adept argues that driving software through the user interface keeps the user in the loop in situations where APIs are missing or incomplete^[1]. That is a different bet from the API-first view of automation: instead of asking every application to expose a clean interface for machines, the model is asked to work the same interface a person works.

Limits

It shipped as a raw base model, without instruction tuning, post-processing or output filtering^[1][2]. A base model is trained to continue text and answer in the format of its training data; it has not been taught to follow instructions politely or to refuse requests, so it is a starting point for further training rather than a finished assistant. The licence is CC-BY-NC 4.0, which rules out commercial use^[1][2]. Taken together, Fuyu-8B is best read as a demonstration of an architecture and a set of priorities — single stack, no encoder, flexible resolution, low latency, screen-shaped inputs — published for others to build on rather than as a product.