Google DeepMind releases Genie 4 world model with audio and real-time steering

Google DeepMind has released Genie 4, its newest world model, as a research preview. According to TechCrunch, the model generates what Google describes as its most photorealistic and interactive worlds yet, and it lets users type prompts mid-simulation to change the scene. It also generates real-time audio that matches the simulated world, including music, ambient noises, and interactive dialogue.

The release lands in a crowded and consolidating field. AI video startup Runway released its second world model, GWM Worlds 2, a couple of months earlier, and the industry is consolidating: Fei-Fei Li’s World Labs is being acquired by AMD, Amazon is in talks to buy Decart, and Rivian has acquired Third Dimension to bring simulation in-house for self-driving testing. Google released the previous model, Genie 3, as a research preview a little over a year ago.

Genie 4’s improvements over its predecessor include far more photorealistic output, audio, and freeform edits such as changing the weather or camera angle in real time or adding new people and objects. It can keep a consistent scene for up to two minutes, double Genie 3’s limit, and it offers better physics and more human-like characters. Google’s Gemini model is also more closely integrated: it acts as the director that plans steps and checks frames while Genie creates the visuals. In a research experiment tied to Gemini Robotics, a Gemini 4 Argon model broke down tasks into steps that Genie 4 simulated in real time, pointing to a possible way to test robot scenarios in simulation before they happen in the real world.

Google research director Tejas Kulkarni describes the direction as hybrid world models: a combination of Gemini writing code and text, and Genie ingesting that in real time and simulating reality at different levels of abstraction. As with other world models, the pitched commercial use cases include gaming, interactive learning, and robotics. One speculative interactive-learning example would be entering a painting of a historical scene and chatting with the painting’s subjects about the portrayed event.

Trusted testers, the people with early access to Google’s cutting-edge products, are also pointing to a possible new use: prototyping films. Google DeepMind engineer Diego Rivas said during a press briefing that real-time generation becomes most valuable in areas where keeping a state of flow in creative work is useful, such as ideation and storyboarding, whenever rapid ideation is needed. Google also offers Veo, an AI video generation model that makes high-definition clips with built-in synchronized audio from text, image, or video prompts and has been growing in popularity among filmmakers. Rivas added that users could eventually rely on Gemini Omni, a multimodal LLM that can generate video from images, audio, and text, to turn those prototypes into creative assets, though those capabilities are still a ways off.

Throughout the briefing, Rivas stressed that it is still early days for Genie 4 and that the end-to-end product experience is not yet clear. The team acknowledged remaining gaps: details and objects can drift the longer a simulation runs, and they have not yet figured out how to ensure audio accuracy and legible in-world text. Genie 4 is available now as a research preview for trusted testers. In the coming weeks, the model will roll out to Google AI Ultra subscribers in Project Genie, Google’s tool for generating interactive worlds from text or image prompts, and to early access partners of the Gemini API and Google AI Studio.

Google’s Genie 4 world model adds audio, more realistic worlds, and real-time steering | TechCrunch

View Original