Why Characters Drift Across Shots
Text-to-video generation is sampling, not casting. Every shot is drawn from a probability distribution conditioned on your prompt, the seed, the motion parameters, and whatever reference data the model can see. A character described only in words — "a woman in her thirties with auburn hair, freckles, and a grey wool coat" — is re-imagined from scratch on every generation. The model has no persistent memory of the woman it drew three shots ago. It has a text string.
That is why consistency collapses in predictable places. The third or fourth shot in a sequence is usually where drift becomes obvious, because the viewer has had time to build a mental model of the face. Hairlines shift, jaw width changes, eye color warms or cools, freckles disappear, a coat's buttons migrate or vanish entirely. Under fast motion, the temporal attention window inside the model is short, so texture detail warps and resettles between frames. At cut boundaries, identity resets almost completely.
The practical cost is high. A marketing spot with an inconsistent spokesperson reads as sloppy. A narrative short loses immersion the moment the protagonist's face changes. Animation pipelines stall because every re-roll burns compute and time without guaranteeing a better result. Consistency is not a cosmetic preference — it is the difference between a set of clips and a sequence that feels authored.
The three sources of drift
- Text-only identity. Descriptive words are lossy. Two creators writing the same sentence will get different faces, and the same creator will get a different face on a re-roll.
- Frame-local generation. Most models reason over short temporal windows, so detail is reconstructed rather than remembered.
- Prompt mutation. Small wording changes between shots — a new adjective, a reordered clause — are interpreted as identity changes.
Multi-image fusion attacks all three at once by giving the model something more stable than language: visual anchors.
How Multi-Image Fusion Works Under the Hood
Multi-image fusion is not image averaging, compositing, or a simple face swap. It is a conditioning strategy. A vision encoder reads several reference images of the same subject and maps them into a compact identity representation — an embedding that encodes geometry, proportions, skin tone, hair structure, and material details like fabric weave or metal trim. That representation is then injected into the generation pipeline, typically through adapter layers or cross-attention controls, so the denoising process is steered toward the same identity manifold in every frame.
The important architectural idea is separation. Identity and style are handled as distinct signals. Identity comes from the reference images; style comes from the prompt, a style reference, or a separately trained adapter. That separation is what makes it possible to keep the same character while changing the visual treatment — realistic drama lighting in one shot, cel-shaded animation in the next, or a branded color grade across an entire campaign.
From references to a stable identity vector
A useful mental model has four stages:
- Feature extraction. Each reference image is encoded, and the encoder focuses on identity-bearing regions: face structure, hairline, body proportions, signature accessories.
- Fusion. The per-image features are merged into a single identity vector. Conflicting information is resolved heuristically — which is exactly why contradictory references produce unstable results.
- Conditioning. The identity vector is injected into the generation process alongside your prompt and any keyframe control.
- Temporal propagation. Across frames, the conditioning keeps re-anchoring the subject so small deviations do not compound into a different person.
Why this beats prompt engineering alone
Prompt engineering sets intent. Fusion supplies evidence. When a model receives both a description and several images of the same person, the images win most conflicts, and the prompt's job shifts to directing action, camera, and lighting instead of policing facial features. That inversion is what turns AI video from a slot machine into a controllable pipeline.
Building a Reference Pack That Holds Up
Most consistency failures are reference failures, not model failures. A pack of five near-identical selfies gives the encoder no information about how the character looks in profile, how their body proportions read in a wide shot, or how their coat behaves when they move.
The minimum viable pack
Aim for four to eight images per character, and prioritize coverage over quantity:
- One clean frontal portrait in neutral light — the anchor image.
- One three-quarter view, which is the most common angle in real footage.
- One profile or near-profile shot to lock nose, jaw, and ear structure.
- One full-body shot in the exact wardrobe used in the sequence.
- One expressive shot (smiling, angry, mid-speech) to teach the model how the face deforms.
- One shot under the dominant lighting of the sequence — warm practicals, cool daylight, or hard key.
For characters who change costumes, build a separate reference group per costume and label them clearly. Mixing wardrobe states in one pack is one of the fastest ways to make a model average two outfits together.
What to exclude
- Heavy occlusion — hands over the face, sunglasses, scarves.
- Strong perspective distortion or wide-lens close-ups that stretch the face.
- Beauty filters, heavy grain, or compression artifacts that the encoder may treat as anatomical features.
- References of different ages, weights, or hairstyles unless you want the model to blend them.
- Conflicting color information. If half your references are in golden-hour light and half in fluorescent light, the model will guess at skin tone, and the guess will fluctuate.
Quality checks before you generate
View the pack as a contact sheet at thumbnail size. If you can still tell it is the same person when the images are tiny, the pack is coherent. If the thumbnails look like siblings rather than the same individual, fix the pack before spending time on video. Resolution matters too — 1024 pixels on the long edge is a reasonable floor for identity-bearing detail; higher is better for full-body references.
Step-by-Step: A Consistent Sequence Workflow
This workflow scales from a single 20-second short to a multi-scene narrative. It assumes a model or tool that supports reference-image conditioning and image-to-video generation.
Step 1: Write a character bible
Before touching a generator, lock the facts. Write a short block that never changes: name, age range, hair color and texture, eye color, skin tone, distinguishing marks, and a wardrobe description with specific materials and colors. Note a palette in hex values if the character is central to a brand. This block becomes a verbatim string you paste into every prompt, untouched.
Step 2: Generate or collect the hero portrait
Create several candidate still images — or gather existing photography — and choose one as the hero. Everything else in the sequence will be judged against this image. Do not proceed until the hero portrait is exactly right; a mediocre anchor guarantees a mediocre sequence.
Step 3: Assemble and validate the reference pack
Fill out the views described above. Run the thumbnail test. Fix contradictions now rather than mid-production.
Step 4: Storyboard as stills first
Generate still keyframes for every shot in the sequence before generating any motion. This is the single highest-leverage habit in the entire workflow. Stills are fast, cheap to iterate, and reveal identity drift instantly. Motion hides drift behind movement, which is why problems seem to appear only after you have rendered a dozen clips.
Step 5: Generate short clips from keyframes
Convert each approved still into a clip of three to five seconds. Longer clips give the model more chances to drift and more opportunities for motion artifacts. Use motion strength settings conservatively at first; aggressive motion values push the model away from the reference conditioning. Keep the seed stable across shots when you want maximum continuity, and vary it deliberately when you want variation in camera energy rather than identity.
Step 6: Assemble and review in context
Cut the clips together and watch the sequence twice — once at full size for craft, once at roughly a quarter scale. Shrinking the frame removes detail bias and makes identity mismatches pop out immediately.
Step 7: Repair per shot, never per sequence
When a shot drifts, regenerate that shot only. Re-anchor it by reusing the keyframe plus the reference pack, and shorten the clip if the drift appears late. Re-rendering the entire sequence resets every solved problem and multiplies inconsistency risk.
Step 8: Version and archive
Keep the approved reference pack, prompt block, seeds, and settings with the finished sequence. If you return to the character later, this record is what makes reproducibility possible.
Keyframe Control and Automated Storyboarding
Keyframes are the bridge between identity conditioning and motion. When you supply a first frame that already contains the correctly conditioned character, the model's job narrows to animating rather than inventing. Start-frame and end-frame interpolation is particularly effective for shots with a predetermined destination: a character turning to camera, a door opening, a product reveal.
Practical habits that pay off:
- Separate subject direction from camera direction. Describe the character's action in one clause and the camera move in another. Blending them confuses conditioning and produces facial stretching during pans.
- Use shot cards. One card per shot with columns for duration, camera, action, wardrobe, lighting, and approved keyframe file name.
- Automate the boring parts. Prompt templates, filename conventions, and contact-sheet assembly can all be scripted or managed in a spreadsheet. Automation reduces the small prompt variations that cause identity drift.
- Name files deterministically. Scene, shot, version, and status in the filename means you never mistake a rejected take for an approved one.
For longer projects, a shot list doubles as a QC instrument. When a character appears in twelve shots, the shot card tells you exactly which keyframe to compare against when something looks off.
Style Changes, Genres, and Edge Cases
Consistency requirements differ by format, and the reference pack should be tuned accordingly.
Cinematic narrative
Photorealistic drama depends on skin texture, eye detail, and micro-expressions. Use high-resolution references under the sequence's dominant lighting. Keep motion moderate. Prefer many short shots over a few long ones — cinematic language already cuts frequently, so short clips never feel choppy.
Animation and stylized sequences
When the character is illustrated or the sequence moves between styles, keep identity references and style references separate. Lock the character with photographic or illustration references, then apply style through prompt language or a dedicated style adapter. If you fuse stylized references into the identity pack, the model may bake the style into the character, making it impossible to switch treatments later.
Marketing and branded content
Brand consistency is character consistency plus palette consistency. Freeze brand colors as part of the identity block, and include wardrobe references that match the actual product. A spokesperson whose outfit shifts tone between shots undermines the whole campaign, even if the face never changes.
Action, crowds, and non-human characters
Fast action reduces the model's ability to hold identity; compensate with more references, shorter clips, and more keyframe control. In crowds, condition the hero character strongly and let background figures stay generic — trying to keep five characters consistent in one shot is an order of magnitude harder than one. For animals, creatures, or mascots, supply references that cover gait and silhouette, since proportion matters as much as face.
Prompting Patterns That Reinforce Identity
Even with strong conditioning, prompts steer the outcome. Use a stable template and change only the variables:
[verbatim identity block] + [wardrobe] + [action] + [camera] + [lighting] + [style]
The identity block never changes. The wardrobe line changes only when the costume changes. Action, camera, lighting, and style carry shot-to-shot variation. Keep the subject at the front of the prompt — early tokens tend to carry more weight.
Do
- Repeat the anchor phrase word for word across every shot.
- Describe wardrobe with materials and colors: "charcoal wool coat, brass buttons, black leather gloves."
- Specify lighting deliberately; changing light changes perceived identity more than most creators expect.
- Use negative prompts for artifacts you consistently see, such as warped hands or duplicated limbs.
Do not
- Re-describe the face with new adjectives each shot. "Sharp cheekbones" in one prompt and "soft features" in the next fights your own reference pack.
- Stack contradictory style words — "photorealistic" and "watercolor" in the same prompt produces an unstable average.
- Rely on negation for identity. "Not blue eyes" rarely works; state what you want and let the references carry the rest.
Quality Control: Catching Drift Early
Build a review pass into every sequence rather than eyeballing the final render. A short checklist catches most problems before delivery:
- Compare shot one against the final shot side by side at identical crop and scale.
- Check the four high-drift zones: hairline, jaw width, ear shape, and eye color.
- Verify wardrobe continuity — collar, buttons, sleeve length, accessory placement.
- Watch at reduced scale to catch structural mismatches that detail distracts you from.
- Freeze-frame at every cut to catch single-frame morphs.
- Check how the character reads under each distinct lighting setup.
If you want a quantitative signal, compare face embeddings between rendered frames and your hero portrait. A falling similarity score across a sequence tells you drift is accumulating even when no single frame looks wrong. Log the score with your shot cards so future productions inherit the knowledge.
Common Mistakes and How to Fix Them
Too many conflicting references. More is not better past a point. Fix: cut to the smallest coherent set that covers the angles you need.
References from different eras of the character. A youthful photo mixed with a current one produces an averaged, uncanny face. Fix: one age and one look per pack.
Skipping the still-frame storyboard. Fix: always approve keyframes before animating.
Re-rendering the whole sequence for one bad shot. Fix: repair per shot.
Changing aspect ratio mid-sequence. Fix: lock composition and ratio for the entire sequence, then adapt for delivery platforms afterward.
Switching models mid-project. Different models interpret the same references differently. Fix: pick one model per character and stay with it, or rebuild the reference pack specifically for the new model.
Over-long clips. Fix: three to five seconds, cut more often.
Ignoring lighting continuity. Fix: treat lighting as part of identity, not decoration.
No version control. Fix: deterministic filenames plus archived reference packs and seeds.
Trusting text alone. Fix: if a character matters, they need images.
FAQ: Character Consistency Questions
Do I need to train a custom model? Usually not. Reference-based conditioning handles most recurring characters. Training a dedicated adapter becomes worthwhile when a character appears across many projects or needs extreme fidelity under heavy motion.
How many reference images is enough? Four is workable, six to eight is comfortable. Below four, expect frequent drift on angles the pack does not cover.
Can I get by with a single image? Yes for short shots from similar angles. Expect weakness in profile, wide shots, and strong expression changes, because the model has to invent everything it has not seen.
Why does the face change when the character turns? The pack probably lacks profile coverage, so the model hallucinates the unseen side. Add three-quarter and profile references.
Does a style transfer break identity? It can. Keep identity and style signals separate, and re-check the hero portrait after any style change.
How long should each clip be? Three to five seconds is the reliability sweet spot. Extend only when the shot genuinely needs it and you have verified consistency across the full length.
What about hands and fine details? Treat them as a separate QC pass. They fail for different reasons than faces and are usually fixed with negative prompts, tighter framing, or slower motion.
Can consistency carry across projects? Yes, if you archive the reference pack, identity block, seeds, and settings. Reproducibility is a documentation habit more than a technical one.
Where to Start Tomorrow
The shortest path to consistent AI video is not a better model — it is a better reference workflow. Write the character bible, build a coherent four-to-eight image pack, approve stills before you animate, keep clips short, and repair drift one shot at a time. Do that consistently and multi-image fusion stops feeling like a trick and starts behaving like a production tool: predictable, repeatable, and fast enough to iterate on creative ideas instead of fighting the generator.
