How AI video models actually work

You type a sentence, wait a minute or two, and a short clip appears: camera moves, a face turns, rain falls. The output looks like photography, but nothing was photographed. Between the sentence and the clip sits a stack of learned components, and knowing roughly what each one does changes how you write prompts, how you judge a bad take, and how much you trust a good one.

This is not a guide to any single product. It is a plain-language tour of the machinery that most commercial video generators share — with the caveat that vendors keep the details private and the field moves fast.

It starts with noise, not with pictures

A diffusion model is trained in two directions. First, a real image is progressively destroyed by adding noise until nothing recognisable remains. Then a neural network is trained to undo that destruction one small step at a time. At generation time the process runs backwards: start from pure noise and denoise repeatedly until an image appears.

That was the design laid out by Ho, Jain and Abbeel in 2020, and it explains a lot about how these tools behave. The model does not retrieve an image from a library. It has learned a distribution over what images look like, and it walks from randomness toward a plausible sample. Ask for the same thing twice and you get two different takes, because the starting noise differs. The model also has no memory of your previous attempt; each generation is a fresh walk.

Most of the work happens in a smaller space

Full-resolution pixels are expensive to generate, so modern systems rarely work at pixel level. The latent diffusion approach introduced by Rombach and colleagues compresses an image with a pretrained autoencoder into a compact latent representation, runs the denoising process there, and decodes the result back to pixels. The same paper added cross-attention layers, the mechanism that lets a text description steer the denoising.

The practical consequence is a division of labour: one network handles compressed structure, another handles language, and a decoder does the final render. The part you influence most directly — the text encoder — talks to the model through attention, not through instructions a human would recognise.

The transformer took over the middle

For years the denoiser was a U-Net. Peebles and Xie replaced it with a transformer operating on latent patches — the Diffusion Transformer, or DiT — and showed that quality improved predictably as compute increased. Since then most large systems have moved in that direction.

One well-documented example of where this went is Stable Diffusion 3, whose paper describes training with rectified flow, a formulation that connects data and noise along a straighter path, and a transformer architecture that keeps separate weights for image and text tokens while letting information flow both ways. The authors report predictable scaling within the range they tested.

Video adds a dimension, and the dimension is expensive

Extending diffusion from images to video is conceptually simple: predict the same kind of noise, but for a sequence of frames. Ho and colleagues did exactly that in a 2022 paper, training jointly on images and video and reporting that the mixture reduced the variance of training updates and sped up optimisation. They also introduced conditional sampling for extending video in space and time — the ancestor of the extend-and-upscale workflow you see in tools now.

The hard part is coherence over time. Every frame can be individually convincing while the sequence still fails: a face drifts, a jacket changes colour, a hand dissolves into the background. What fixes this is attention across frames — the model has to see the character in shot one while drawing shot two. When people talk about "temporal consistency", that is the mechanism they mean.

The prompt only reaches the model through language

Text is encoded into embeddings and the denoiser attends to them while drawing. This is why prompts that name visible things outperform prompts that name feelings. "A man is sad" gives the model almost nothing to draw. "A man alone at a kitchen table, head down, cold window light from the left, medium shot, static camera" gives it a subject, a staging, a lens and a mood.

It also explains a common frustration: the model is not following directions in the way a camera crew would. It is being nudged, frame by frame, toward a region of its training distribution that your words and reference images point to.

Sound is now inside the model

Until recently you generated silent clips and built the soundtrack in post. In 2025 Google's Veo 3 became the company's first video model to generate dialogue, sound effects and music synchronised with the picture in a single pass. Runway's multi-shot recipe likewise generates audio by default. Native audio is genuinely new, and it changes planning: a spoken line inside a generated clip is a performance choice you make at prompt time, not in the edit.

It also raises the bar: a shot with native sound has to be judged on lip-sync, room tone and intelligibility, not just on how pretty the frame is.

What the model is not

Five limits are worth keeping in mind, because each one maps to a different failure you will meet in practice:

• Not a physics engine. It predicts plausible pixels. Contact, weight, cloth and fluids are learned habits, not simulation, which is why falling objects and hands that grip still break.
• Not a camera. Camera language works as a suggestion, not a cue sheet. Expect to iterate on the move rather than set it once.
• Not deterministic. The same prompt yields different results. Reproducibility lives in seeds, fixed references and pinned model versions, not in your prose.
• Not a database. It cannot retrieve your actor's face or your actual location unless you condition it with reference images or a first frame.
• Not a rights-clearance service. A model that can draw something does not tell you whether you may use it. Training data, likeness and output rights are separate questions with their own answers.

Why the plumbing matters on set

Once you see the pipeline as a sequence of learned stages, the workflow follows. You direct shots instead of scenes, because a shot is the unit the model can hold. You generate many cheap variations and select, because sampling is the only way to see the distribution. You lock references for anything that must persist between shots, because consistency is an attention problem, not a wording problem.

The practical version of this article — how to write the shot, what to feed the model, how to fix the take you almost got — is in the AI video generation tips. For how these techniques show up in finished work, see the guide to AI-generated short films.