TellWorldAI World Archive

Home›Practice›Practice

Practice

Generating interactive worlds with Genie 4 in the Gemini API

How to open a Genie 4 session in the Gemini API, stream frames and audio, steer the world with keys and text, and end sessions to stop billing.

If you want a prototype where the user walks through a scene that is generated live, controlled by the keyboard, this guide shows how to open a Genie 4 session, connect to it from Python or a browser, send the first world setup and movement inputs, and end the session so billing stops.

What Genie 4 is

Genie 4 uses the model name models/genie-4. Through the Interactions API it produces a continuous, interactive 16:9 world at 24 fps, with 48 kHz stereo audio synchronized to the picture [1]. It differs from Gemini Omni 1.2 Flash, which generates offline clips. Genie 4 renders each frame on the fly in response to live keys (WASD and the arrow keys) and to natural-language events sent during the session [1]. Offline clip generation is the job of Gemini Omni 1.2 Flash [1].

What you need and what it costs

Genie 4 requires a paid-tier Gemini API key [1]. Billing is per video output token: 240 tokens per frame, which is 5,760 tokens per second at 24 fps, or about $0.10 per second with stereo audio included [1]. A session keeps billing while it is open, so end every test session explicitly.

Location matters for responsiveness. Genie 4 serves from data centers in the United States, Western Europe and Japan. Round-trip latency under 150 ms is expected in North America, Western and Central Europe and East Asia. Elsewhere, steering lags and the frame rate can drop, so place any backend relay near a serving region when you can [1]. Pricing, rate limits and specifications are listed on their own pages [1].

The four-step lifecycle

Every session follows the same four steps on v1beta [1]:

  1. Create the interaction on your server.
  2. Connect a live session. Browser apps use WebRTC, and Python, Node.js or other browser apps use WebSocket.
  3. Send one world_setup event to set the scene, then steer with user_action or input events.
  4. Call complete() when the session is done.

Browsers must never receive a long-lived GEMINI_API_KEY. Create the interaction on the backend, mint a single-session token with interactions.authTokens.create, and hand only that token and the interaction ID to the client [1]. The Python example below splits the work between the backend and the client:

from google import genai

# Backend: create interaction and mint single-session token
server_client = genai.Client(http_options={"api_version": "v1beta"})
interaction = await server_client.aio.interactions.create(
    model="models/genie-4", api_version="v1beta"
)
token_resp = await server_client.aio.interactions.auth_tokens.create(
    interaction_id=interaction.id, api_version="v1beta"
)

# Client worker: authenticate solely with the ephemeral session token
scoped_client = genai.Client(
    api_key=token_resp.auth_token,
    http_options={"api_version": "v1beta"},
)
async with scoped_client.aio.interactions.connect(
    interaction.id, api_version="v1beta"
) as session:
    ...

What the first run looks like

Initial world setup takes roughly 5 to 9 seconds before the first frame arrives [1]. A text change made mid-session, such as snow falling over the village, takes about 1 to 2 seconds to appear, because Genie 4 plans environmental changes between generation steps [1].

Its spatial memory is short. Genie 4 keeps a rolling visual horizon instead of a persistent 3D scene. If you turn away for a few seconds and turn back, background buildings may have been rearranged [1].

Setting the scene with world_setup

Send one world_setup event at the start of each session. It can carry a text prompt, a 16:9 starting image, or both [1]. A text-only version looks like this:

{
  "event_type": "world_setup",
  "world_prompt": {
    "text": "First-person POV standing in front of an espresso machine. An empty ceramic mug rests on the drip tray beneath the spout. A glowing \"brew\" button is visible on the front panel."
  }
}

Moving and looking with user_action

Movement and camera control use user_action. The action field takes key_down, key_up or key_click. Send key_down when the user presses a key, and do not forward the operating system's auto-repeat events. Send key_up on release. A held key keeps applying on each step until key_up arrives [1].

{
  "event_type": "user_action",
  "action": "key_down",
  "input": "key_w"
}

The built-in movement keys are key_w, key_s, key_a and key_d. The camera keys are arrow_up, arrow_down, arrow_left and arrow_right [1]. The protocol also accepts key_e and space, but Genie 4 attaches no jump or interact behavior to them. For custom verbs such as opening a door, intercept those keys in the client and send a descriptive input text event instead [1].

Changing the scene with text: input

Text input events change the environment, object states or character dialogue while the stream runs. World planning reads text at step boundaries, and a step covers 4 frames, which is about 0.17 seconds at 24 fps. A one-off instruction therefore works best when you send it 10 times at 150 ms intervals, about 1.5 seconds in total, so the next planning window catches it. Leave a cooldown of about 2 seconds before sending a conflicting prompt [1].

Ending a session: complete, cancel and detach

These three operations have different billing effects. complete() tells the server to finish the interaction, stops generation and billing, and emits a final interaction.completed event on receive(). Call it whenever a user finishes [1]. cancel() aborts server-side generation immediately [1].

detach(), or leaving Python's async with block, only closes the local connection. The server-side interaction is still open. Genie 4 does not support session resumption, so detaching cannot pause the session or stop billing [1]. Closing a browser tab behaves the same way: the transport detaches, but generation does not stop right away [1].

Decoding the audio and video stream

Over WebSocket, picture data arrives in step.delta events. Video deltas have the type video and carry one base64-encoded JPEG frame each. Audio deltas have the type audio and carry base64-encoded raw 16-bit little-endian stereo PCM at 48,000 Hz [1]. Decode the audio with the right sample rate, sample width and channel count, or it will sound distorted.

Limits

  • Response time: native keyboard actions take effect in about 0.2 to 0.5 seconds, and text prompts take about 1 to 2 seconds [1].
  • Output format: the output is fixed at 1280x720 and 24 fps in 16:9. Portrait 9:16 and custom frame rates are not supported [1].
  • Inputs: Genie 4 does not accept live microphone or camera input, and it does not support function calling, Google Search grounding, structured outputs or session resumption [1].
  • Spatial memory: looking at an object for 8 seconds or more, or turning 360 degrees, can move or reset landmarks, doors and buttons. Visual stability may degrade past 60 to 90 seconds, with a hard cap at 120 seconds [1].
  • Content filters: prompts or starting images with identifiable real people, public figures or copyrighted brand IP can be blocked at startup, or can end a running stream [1].
  • Details: signs, gauges and clocks deform over time, and object counts drift when the camera pans [1].

Where to go next

Before building on Genie 4, test steering latency and scene continuity with a small scene. Check the model specifications, pricing and concurrent session limits on the official pages, since those figures change more often than this guide [1].

Sources

  1. 1

Cite this page: Generating interactive worlds with Genie 4 in the Gemini API. TellWorld, https://www.tellworld.net/practice/practice-genie-4-api. Updated 2026-11-20

Related news

View all