The Text-to-Image article covered generating a single, static picture. This article covers the natural, much harder next step: generating an entire sequence of moving images from a written description, called text-to-video.
The simple definition
Text-to-video generates a moving video clip from a written description, extending the diffusion process covered throughout this phase across time as well as space. Recall from the Diffusion Model article’s noise-removal process, applied to a single image. Text-to-video applies essentially the same underlying idea, but now has to solve a genuinely harder problem: not just generating one coherent frame, but generating many frames in sequence that stay consistent with each other — the same character’s face, the same object’s shape, staying stable and coherent as the “camera” and subjects move.
Why this is a genuinely much harder problem than a single image
Recall from the Text-to-Image article’s controversy over whether a single generated image counts as art. Text-to-video has to solve that same generation problem, repeated dozens of times per second, while also solving an entirely new problem the single-image case never faced: temporal consistency.
+A generated video needs an object’s appearance, lighting, and physical behavior to remain coherent from one frame to the next — recall from the Sequence article, back in the Data Representation phase, that order and consistency across a sequence carry real meaning, and a text-to-video model has to maintain that consistency across an entire video’s worth of generated frames, not just get one single frame right.
flowchart LR
A["Text prompt: 'a dog running on a beach'"] --> B[Generate a coherent sequence of frames]
B --> C[Frame 1: dog mid-stride] --> D[Frame 2: dog's next stride, same dog, same beach]
D --> E["...consistent across the entire clip"]The real, genuinely funny before-and-after story that tracks this technology’s progress
This is worth telling in full, because it’s one of the most widely shared, informally used benchmarks in the entire field — genuinely useful precisely because it’s so easy for anyone to judge for themselves. On March 23, 2023, a Reddit user posted a video generated using ModelScope, an early text-to-video tool, depicting actor Will Smith eating spaghetti.
+The result was, by every account, a disaster — Smith’s face morphed unpredictably between mismatched expressions, his hands became rubbery, distorted appendages, and the noodles appeared to float with their own strange, independent gravity.
+It went viral specifically because it was so uncannily bad, becoming informal shorthand across the AI community for “the current, awkward, adolescent state of text-to-video technology.” In February 2024, Will Smith himself joined the joke, posting his own real, live-action parody video on social media, captioned “This is getting out of hand!” Then, over the following two years, as text-to-video models genuinely improved — OpenAI’s Sora, Google’s Veo, China’s Kling and MiniMax — people kept regenerating the exact same “Will Smith eating spaghetti” prompt as an informal progress check.
+By 2025, Google’s Veo 3 produced a version with realistic chewing and slurping sounds, natural hand movement, and convincing facial expressions — and by the time Kling 3.0 arrived, the same basic prompt could produce an entire coherent scene of Smith eating spaghetti while having a conversation with a child, all from one single prompt.
ANALOGY vs. TECHNICAL REALITY
Analogy: Think of a flip-book animator who has to draw not just one convincing picture, but hundreds of pictures in a row, each one subtly different from the last, all needing to stay consistent enough that flipping through them quickly produces smooth, believable motion rather than a jarring, flickering mess.
Where this breaks down: A flip-book animator draws each frame with conscious continuity in mind, referring back to earlier pages. A text-to-video model generates its frames through the same statistical, learned diffusion process covered throughout this phase, with temporal consistency achieved through specific architectural techniques — extending attention mechanisms, covered throughout the Transformers phase, across the time dimension as well as the spatial one — rather than a human-style conscious effort to “remember” what the last frame looked like.
Why the Will Smith test genuinely mattered, beyond being a meme
It’s worth being direct about why this informal benchmark carried real, substantive weight. Human faces and hand movements are notoriously difficult for generative models to get right — small errors that would be invisible in a landscape or an abstract scene become immediately, viscerally obvious on a recognizable human face performing a familiar, everyday action. A model succeeding at “Will Smith eating spaghetti convincingly” was a genuinely meaningful signal of real technical progress on exactly the hardest, most scrutinized category of content, not an arbitrary or frivolous test.
The real, published specs behind today’s leading systems
This is worth grounding in actual, documented numbers rather than vague claims of “impressive quality.” OpenAI’s Sora, in its full model form, can generate videos up to 60 seconds long at 1080p resolution, supporting camera movements like panning, zooming, and tracking shots, and accepting not just text but also image and existing-video inputs to guide generation — directly overlapping with the image-to-image techniques covered in the next article, just extended across time.
+Its 2025 successor, Sora 2, added a genuinely significant capability: synchronized audio generated together with the video itself — dialogue, sound effects, and ambient noise produced in the same pass as the visuals, rather than added afterward as a separate step.
+Real, current access is tiered: ChatGPT Plus subscribers get a limited number of shorter, lower-resolution videos per month, while Pro subscribers and the dedicated Sora platform unlock longer clips at higher resolution — a genuine, practical illustration of how computationally expensive this technology remains even now.
A concrete example, layered
For a simple beginner example: a marketing team might generate a short video of “a coffee cup steaming on a wooden table, morning light” for a product ad’s background footage, entirely from a text description, without ever filming anything. For a production example: OpenAI’s Sora, publicly demonstrated and later released, and Google’s Veo 3, both real, published text-to-video systems, represent the current frontier of this technology — with Veo 3’s own real-world “spaghetti test” results specifically cited by tech journalists as evidence of how far the field had come from the original 2023 ModelScope clip in just two years.
Why this technology raises the same, and new, real concerns
It’s worth connecting this directly back to the GAN article’s deepfake concerns. Text-to-video inherits every one of the misinformation and consent risks GANs first raised, now applied to fully synthetic, moving footage of real people rather than static, GAN-generated faces — a genuinely more visceral, harder-to-dismiss category of concern, actively shaping how companies like OpenAI and Google build in safeguards, watermarking, and usage restrictions around their text-to-video products.
Why good individual frames are not enough
Suppose a model creates four individually attractive frames:
Frame 1: a red car with four doors
Frame 2: the car suddenly has two doors
Frame 3: its color changes to orange
Frame 4: the road moves in an impossible directionEach frame might look realistic when viewed alone, but the video fails because objects and motion do not remain consistent through time. This is called a temporal consistency problem.
A video model must coordinate several things:
- Object persistence: the same person, animal, or object should remain recognizable.
- Motion: bodies, cloth, water, and vehicles should move plausibly.
- Camera movement: panning, zooming, and viewpoint changes should follow a coherent path.
- Scene continuity: lighting, background, and object locations should not jump unexpectedly.
- Audio synchronization: when audio is generated, speech and sound should match visible events.
Real text-to-video applications
Google’s Veo 3.1 can generate video from a text prompt, and its workflow can also use an image as a starting frame. A marketing application might submit, “A slow camera move across a miniature solar-powered city at sunrise”, then poll the long-running generation job until the video is ready.
OpenAI’s documented Sora video endpoint illustrates the same asynchronous pattern:
prompt → create video job → queued → in progress → completed or failed
↓
progress percentageThe documented Sora endpoint accepts sora-2 or sora-2-pro, optional reference imagery, and clip lengths of 4, 8, or 12 seconds. However, the official documentation marks this API as deprecated and scheduled to shut down on September 24, 2026, so it is a useful concrete example of video generation but not a safe foundation for a new long-lived integration. Sources: Google’s Veo 3.1 guide and OpenAI’s video creation reference.
Common misconception
A frequent beginner assumption: that current text-to-video technology can already reliably produce broadcast-quality, feature-length content on demand. As the Will Smith story’s own multi-year timeline demonstrated, this remains a genuinely fast-moving but still-maturing technology — even now, longer clips, complex multi-character interactions, and precise physical accuracy remain real, active areas of ongoing improvement, not a fully solved problem.
Where this fits in what comes next
You now understand generating video from text, and how publicly, dramatically this specific capability has improved. The next article, Image-to-Image, covers a related but distinct generative task — transforming an existing image into a new one, rather than generating either an image or video from text alone.
In one sentence
Text-to-video extends the diffusion process across time as well as space, solving the much harder problem of keeping generated frames consistent with each other — and its remarkably fast, publicly visible progress is best captured by the “Will Smith eating spaghetti” test, which went from a viral, uncanny disaster in 2023 to a convincing, coherent scene within just two years.
Related Terms
- Author
- TechByteByByte Editorial Team
- Reviewed by
- TechByteByByte Admin
- Published
- Last reviewed