Text-to-video models have become remarkably good at producing one beautiful clip. They remain mediocre at producing ten clips that look like they belong to the same film. That gap between a single impressive shot and a coherent sequence is where most AI video projects stall, and it is the gap that multi-modal conditioning closes.
The shift is conceptual as much as technical. A prompt is a brief. Reference images, depth maps, pose guides, and audio are constraints. Once you start treating generation as a matter of stacking constraints instead of writing longer sentences, photorealistic sequences stop being a matter of luck.
Why Text-Only Prompts Hit a Ceiling
A prompt like "a woman in a red coat walks through a rainy Tokyo street at night, cinematic" is not a specification. It is a mood board compressed into a sentence. The model has to resolve dozens of open variables: the cut and fabric of the coat, the focal length, the colour temperature of the streetlights, the density of the rain, the height of the camera, the pace of the walk, the amount of grain. Every generation re-rolls those dice. That is why two shots generated from the same prompt rarely cut together.
The deeper problem is that language encodes meaning, not geometry. Text can say "she turns her head." It cannot say that the key light falls three centimetres below her cheekbone, that the camera is 1.2 metres from her collar, or that her shoulder blocks the neon sign behind her. Continuity is a property of the physical world. Prompts describe the world abstractly, and abstraction is exactly where drift creeps in.
There is also a physics problem. Models trained on short clips learn plausible motion, not consistent motion. Rain falls at slightly different speeds, a coat swings with different mass, a footstep lands with different weight. Human eyes are tuned to detect these mismatches instantly, even when viewers cannot articulate what feels wrong. The result is the familiar uncanny shimmer that makes an otherwise gorgeous clip feel synthetic.
The fix is not a better vocabulary. It is a richer input stack: images that pin down identity, style plates that pin down grade, structural guides that pin down geometry, and text reduced to the layer it handles best — intent.
What Each Input Type Actually Controls
The most common mistake in multi-modal generation is assuming every input affects everything. They do not. Each modality has a narrow, predictable zone of influence, and knowing those zones lets you fix problems surgically instead of rewriting the whole prompt.
Reference images lock identity and wardrobe
A single well-lit reference image is the most powerful continuity tool available. It fixes facial structure, hair, skin tone, and garment design far more reliably than any adjective. Two or three references from different angles work better than one, because the model can triangulate three-dimensional structure instead of guessing at the back of a head it has never seen.
Resolution matters more than quantity. A 1024-pixel reference with clean, even lighting outperforms five blurry phone photos. If your character sheet has hard shadows across the face, those shadows will follow the character into every scene, including ones lit from the opposite side.
Style plates and colour scripts carry grade and texture
Style transfer is often used as a blunt instrument, but its real value is consistency. A single frame from a film you admire, used as a style reference, communicates more about contrast, saturation, grain, and highlight roll-off than a paragraph of cinematography vocabulary. Pair it with a simple three-colour script — midtones, shadows, highlights — so that every shot in a sequence shares the same palette.
Keep style references separate from identity references. Mixing them in one input slot forces the model to average the two, which produces faces that look like nobody and colours that look like nothing.
Depth, normal, and pose maps carry geometry and motion
Structural guides are the least glamorous and most underrated inputs. A depth map tells the model how far away every surface is, which stabilises parallax and prevents backgrounds from sliding around. A pose map, usually derived from a human skeleton tracker, controls body mechanics so a walk cycle does not turn into a shuffle. Normal maps add surface orientation, which helps with skin shading and fabric folds.
You do not need all three at once. Start with depth for environments, add pose for characters, and introduce normals only when surfaces look flat.
Text is the intent layer
Once images and structure are doing the heavy lifting, the prompt shrinks. Instead of describing appearance, it describes action and beat: "she stops, checks her phone, keeps walking." That kind of prompt is short, unambiguous, and hard for the model to misinterpret. Any adjective you keep in the text should be one you genuinely cannot express visually.
Audio sets timing
Audio is an input that most creators add last, and it should often come first. A dialogue track or a music bed gives the model a rhythm to cut against and tells you exactly how long a shot needs to be. Generating to a locked audio timeline eliminates the endless trimming that happens when you discover a clip is 0.4 seconds too short for the beat.
Building a Character That Survives the Cut
Character consistency is the single most common failure point in AI sequences. The good news is that it is largely solvable with a disciplined pipeline.
Create a character sheet before you animate anything
Start with a still-image model and generate a sheet: front, three-quarter, profile, plus one full-body frame and one close-up. Keep the background neutral grey, the lighting even, and the expression relaxed. If the character is costumed, generate the costume on the same body rather than inventing it later in a video model. A costume invented mid-sequence will mutate.
Attach a lightweight identity adapter
For a single short project, image-to-video conditioning with a strong reference is usually enough. For anything with more than four or five shots, a trained identity adapter — a small LoRA-style model, or a face-embedding adapter in a node-based interface such as ComfyUI — pays for itself in saved retries. Train on 15 to 30 varied images, including different expressions and angles, and avoid images with harsh shadows or heavy makeup.
Keep wardrobe and lighting consistent across shots
Wardrobe drift is subtle and devastating: a jacket gains a zipper, a scarf changes colour, buttons move. Solve it by attaching the costume reference to every shot rather than describing it in text. Lighting is a similar trap. If a character is lit from the left in the master shot, they must be lit from the left in the reverse, even if that means overriding the model's default lighting instinct with a relight pass in post.
Structural Directives: Camera, Lens, and Blocking
Photorealism is as much about camera behaviour as it is about skin texture. A perfectly rendered face shot with an impossible lens still reads as artificial.
Translate camera language into parameters
"Heroic low angle" means nothing to a model. Instead, define:
- Focal length: 35mm for dialogue, 85mm for close-ups, 24mm or wider for establishing shots.
- Camera height: eye level, chest level, or knee level, stated explicitly.
- Movement: locked-off, slow dolly in, lateral track, crane up, or handheld with slight bob.
- Shutter and grain: enough motion blur to match your frame rate, plus grain that matches your style plate.
If you have reference footage from a real camera, feed a short clip as a motion reference. Video-to-video conditioning is the most direct way to inherit a real camera's personality, including its lens breathing and its imperfect handheld drift.
Use last-frame chaining for match cuts
Continuing a shot is far easier than recreating one. Take the final frame of clip A, use it as the starting image for clip B, and add a short motion instruction. The model inherits lighting, wardrobe, and grain automatically. Chain three or four clips this way and you can build a genuine long take without visible seams.
The technique has a limit: errors compound. Each chained frame carries forward a little degradation, so check the final clip in the chain before you commit to a five-link sequence.
Control motion with drivers, not hope
When a subject's movement matters, drive it. Animate a simple 3D proxy, export a depth or pose sequence, and use it as the structural guide. This is how you get a specific gesture — a hand reaching for a handle, a dancer's turn — instead of a plausible approximation. It takes an extra twenty minutes of setup and saves hours of re-rolling.
Model Selection Without Breaking Continuity
Different models have different strengths: one renders skin and hair beautifully but drifts on long motion; another handles action and camera moves but softens faces; a third produces gorgeous stills and weak video. Continuity survives model switching only if you keep the shared constraints identical.
Match the model to the beat
Use a photoreal-leaning model for dialogue and close-ups. Use a motion-focused model for chases, crowd movement, and complex camera work. Use a dedicated upscaler for the final pass rather than asking a video model to output at delivery resolution. Write this mapping down before you start, so you are not re-deciding per shot.
Compare models on the same reference frame
When testing a new model, run the identical reference image, style plate, and prompt through it. Comparing outputs generated from different inputs tells you nothing. Comparing the same input across models tells you everything: which one holds the face, which one holds the grain, which one respects the depth map.
Budget your generation passes deliberately
Every project has a finite render budget, whether measured in money, GPU minutes, or evening hours. Spend it where the audience looks. Generate low-resolution proxy passes to solve blocking, timing, and camera movement, then spend full-quality renders only on shots that survive the proxy review. A two-tier pass structure typically cuts total generation time by half and produces better results, because you stop re-rendering shots you were going to cut anyway.
A Repeatable Production Workflow
Here is a pipeline that scales from a thirty-second social clip to a multi-minute narrative short.
Step 1 — Build a shot list with intent
Describe each shot in one line: what the audience must understand, and how long the shot needs to be. This document becomes the spine of the whole project. Anything not needed for comprehension is a candidate for cutting.
Step 2 — Generate the reference pass
Create character sheets, location plates, and one key frame per shot. Review them as a contact sheet. If the stills do not look like a coherent film, no amount of video generation will fix it.
Step 3 — Build an animatic
Assemble the key frames on a timeline with temporary audio and rough motion. This is where pacing problems appear, cheaply. Cutting a still is free; cutting a finished clip is not.
Step 4 — Generate hero shots
Animate shot by shot, in order, using frame chaining where shots are adjacent. Render at proxy resolution first, review, then upscale the keepers. Keep a running continuity log: wardrobe state, props, light direction, time of day.
Step 5 — Assemble and finish
Bring everything into an editor. Stabilise the inevitable micro-jitter, colour-match across models, and add grain as a final unifying layer. One grain pass over the whole sequence hides more model differences than any individual colour correction.
Audio, Pacing, and the Edit
Sound is not decoration; it is the strongest continuity tool in the entire pipeline. A consistent room tone across cuts makes continuity errors far less visible. Footsteps and cloth movement sell weight, which is precisely what AI motion lacks. If a character's heel strike does not land on the beat, viewers register it as fake even if they cannot say why.
Build a scratch track first: dialogue, then foley for contacts and movement, then music. Cut picture to that track. When you generate video, use the audio as a timing reference so that clips arrive at the correct duration rather than being stretched in post. Avoid speed-ramping generated footage, because variable-speed playback exposes frame interpolation artefacts almost immediately.
Finally, cut on motion. Cuts that land mid-movement hide small continuity mismatches; cuts that land on stillness expose them. When two shots refuse to match, look for a moving frame near the seam and cut there.
Common Mistakes That Break Photorealism
- Changing the reference mid-sequence. Even a slightly different reference image triggers a visible identity shift. Lock your references at the start.
- Overloading the prompt. Conflicting style words — "documentary" plus "hyper-stylised" — force the model to average contradictory looks, producing a flat, generic image.
- Using one model for everything. No single model wins at faces, action, and upscaling. Specialise.
- Ignoring frame rate and motion blur. A 24fps cinematic look needs blur; a crisp 60fps look does not. Mismatched blur reads as cheap.
- Upscaling before grading. Upscalers amplify noise and colour shifts. Grade first, then upscale, then add grain.
- Skipping the colour script. Without a palette, each model quietly applies its own default grade, and the sequence looks stitched together.
- Generating clips that are too long. Most models drift after a few seconds. Build long shots from chained short clips instead.
- Forgetting hands, feet, and contact points. Objects that touch — a cup, a doorknob, a chair — are where photorealism usually collapses.
A Practical Quality-Control Checklist
Run every shot through the same list before it enters the timeline:
- Identity: does the face match the reference at 100% zoom in the first, middle, and last frame?
- Wardrobe: are garment details, colours, and fastenings unchanged from the previous shot?
- Lighting: does the key light direction match the adjacent shots?
- Flicker: watch at quarter speed for texture boiling on skin, walls, and fabric.
- Contact: check every hand-object interaction frame by frame.
- Background stability: do distant details stay anchored, or do they swim?
- Motion physics: does weight transfer read correctly, or does the subject glide?
- Grade: does the shot sit inside the colour script without correction?
Anything that fails two or more checks goes back for a re-render with a stronger structural guide.
FAQ
Do I need specialised software to use multi-modal inputs?
No. Most current video tools accept an image plus text at minimum. Node-based interfaces give you the most control over depth, pose, and style inputs, but a simple image-to-video workflow with a good reference already solves most continuity problems.
How many reference images does a character need?
Two or three clean angles are enough for short projects. For recurring characters across many shots, a trained identity adapter built from 15 to 30 varied images is more reliable and faster to reuse.
Why do my clips look sharp but still fake?
Usually a camera problem, not a rendering problem. Missing motion blur, impossible focal lengths, zero camera drift, and perfectly even lighting all signal artificiality. Add blur, choose a believable lens, and let the camera be slightly imperfect.
How long can a single generated clip be before it drifts?
Practically, three to six seconds per generation is the safe zone for most models. Beyond that, faces soften, backgrounds swim, and physics degrade. Chain shorter clips with last-frame handoff to build longer shots.
Should I generate video first or audio first?
Audio first, whenever dialogue or music drives the pacing. Generating to a locked track removes an entire class of timing problems and prevents awkward stretching in the edit.
Can I mix styles across a sequence?
You can, but you must unify them in post. Apply one grain layer, one colour script, and one contrast curve across the whole timeline. Consistency is a finishing decision as much as a generation one.
What is the fastest way to improve results with no extra tools?
Shorten your prompt and attach one good reference image. Most prompt bloat exists to compensate for missing visual constraints, and once those constraints are supplied, the extra words actively hurt.
Photorealistic AI video is no longer a matter of writing the perfect sentence. It is a matter of deciding what each input should control, locking the things that must not change, and spending your render budget on the shots that carry the story. Do that, and the sequence stops looking like a collection of clips and starts looking like a film.
