Why AI characters drift between shots
Text-to-video is generative at every step. When you write a prompt such as a lone desert wanderer in a linen cloak, the model has no stored memory of who that person is. It samples a face, a hairline, a jaw shape, a wardrobe, and a lighting response for every frame, guided only by the words in front of it. The result is a character who looks plausible in isolation and unrecognizable three cuts later.
Drift is rarely random. It clusters around predictable triggers: large changes in camera distance, profile turns, hard changes in light direction, new props entering frame, and long takes where the model has to invent detail it was never shown. Naming those triggers is the first productive step, because it tells you where a pipeline needs anchors instead of hope.
A director working with human actors solves this with continuity: wardrobe notes, reference stills, a script supervisor watching every take. Multi-image fusion is the generative equivalent. Instead of describing a character in words and praying the sampler agrees, you hand the model several images of the same person and let it build a fused identity it can hold across shots.
What multi-image fusion actually changes
Multi-image fusion means conditioning a video model on a set of images rather than a single frame or a pure text prompt. An encoder extracts appearance features from each reference, blends them into one identity representation, and applies that representation during generation. Depending on the tool, that can happen inside the latent space, as a lightweight adapter trained on the fly, or as a per-frame identity injection combined with image-to-video keyframing.
The practical difference is stability. A single reference gives the model one view to extrapolate from, so a turn to profile becomes a guess. Five references that include a three-quarter left, a three-quarter right, and a profile give the model enough geometry to keep the same nose, cheekbone, and hairline when the camera moves.
Identity conditioning versus style conditioning
Identity is who the character is: bone structure, hairline, skin tone, eye shape, distinguishing marks, wardrobe silhouette. Style is how the shot is rendered: lens character, grain, color grade, lighting mood. Mixing the two in one prompt is the most common reason a face changes when the scene changes, because the model has no way to know which tokens should stay fixed. Keep identity tokens in an untouched block and let style tokens move between shots.
More references is not automatically better
Three to six clean, mutually consistent references usually beat twenty messy ones. If your reference set contains conflicting information, the fusion step averages it into a vague face that matches none of your images. Every reference you add should answer a question the model would otherwise have to invent: what does this person look like from the left, in motion, in full body, under neutral light.
Building a reference set that survives camera changes
The quality of your fused identity is capped by the quality of your reference set. Treat it like a continuity kit, not a mood board.
The six-angle character sheet
For each character, aim to collect:
- A neutral front head-and-shoulders frame
- A three-quarter view from each side
- A true profile
- A full-body frame that shows proportions and wardrobe silhouette
- One action or expression frame that shows how the face deforms when the character moves
If you cannot source all six, generate the missing angles with an image model using the frames you trust most, then inspect them critically. A generated profile with the wrong ear shape will poison every profile shot that follows.
Lighting, background, and crop hygiene
Shoot or generate references under soft, neutral light with a simple background. Flat lighting hides form, but dramatic lighting bakes a shadow pattern into the identity, and that shadow will follow your character into every scene. Keep crop and aspect consistent across the set so the encoder is not compensating for framing differences. For head references, the face should occupy roughly forty to sixty percent of the frame with the eyes sharp and unobstructed.
What to leave out
Exclude watermarks, sunglasses, heavy beauty filters, motion blur, group photos, extreme expressions with squinted eyes, and anything with dramatically different makeup or hair styling. Also avoid mixing references from different ages of the same character unless you deliberately want a blended, ambiguous age that will hurt both the young and old versions of the story.
A step-by-step multi-image fusion workflow
This sequence works whether you are generating a single scene or a ten-episode series. The principle is simple: fix identity at the cheapest possible layer before spending time on motion.
Step 1: write the character bible
Describe each character in a fixed order: age band, build, hair color and cut, skin tone, eye color, distinguishing marks, wardrobe layers, and palette. Save this as a reusable text block and paste it verbatim into every prompt. Rewriting the description from memory is how identity quietly shifts between sessions.
Step 2: assemble and normalize references
Crop to a consistent aspect ratio, upscale low-resolution images to the model-native resolution, and give files descriptive names such as maya_front_neutral and maya_profile_left. Delete near-duplicates: two nearly identical frames add no information and dilute the fusion weights.
Step 3: lock keyframes before you animate
Generate still images for each shot first. Choose the frame that best matches your character sheet, fix any obvious errors with an image editor, and only then animate from that still using image-to-video. Iterating on a static frame is dramatically cheaper and faster than re-generating motion, and the model inherits a correct identity for frame one.
Step 4: generate shot by shot with anchors
Keep clips short, typically two to five seconds, with one clear action per shot. Reuse the same seed when the tool supports it, keep the identity block identical, and change only the scene, camera, and motion blocks. Raise reference strength when the face drifts; lower it when the character needs to emote, turn sharply, or move fast, because over-constrained identity can freeze the performance into a mannequin stare.
Step 5: run QA and triage re-rolls
Watch each clip at full speed, then scrub at quarter speed. When something fails, identify which variable caused it before re-rolling. If the face is right but the coat turned blue, that is a wardrobe token problem. If the coat is right and the face shifted, that is a reference or weighting problem. If both are right and the motion looks rubbery, that is a motion-prompt or duration problem. Re-rolling without diagnosis just burns time.
Prompt patterns that keep a face stable
A repeatable prompt skeleton beats clever prose. Keep the identity block first and never reword it.
[IDENTITY] 34-year-old, short black hair swept back, sharp jaw, small scar above left eyebrow, olive skin
[WARDROBE] charcoal linen coat, cream undershirt, leather satchel on right shoulder
[SCENE] coastal road at dusk, low sun from frame left, light mist
[CAMERA] medium close-up, 50mm equivalent, slow dolly in
[MOTION] walks forward, coat moving in wind, steady gaze
[STYLE] natural light, shallow depth of field, subtle film grain
Two rules make this work. First, never use synonyms for identity tokens: charcoal linen coat and dark grey jacket are different clothing to a model, and swapping them mid-project reads as a costume change. Second, keep the block order identical so attention patterns stay stable.
For negative prompts, list the failures you actually saw: different face, changing hair color, warped hands, extra fingers, morphing clothing, text overlay, watermark, flickering background. A generic negative list wastes conditioning capacity on problems you do not have.
When a shot needs a strong expression, consider lowering identity strength and compensating with a reference image that already shows a similar expression. Emotion and identity compete for the same conditioning budget, so giving the model an emotional reference is more reliable than pushing identity weights harder.
Choosing tools that fit a consistency-first pipeline
Before committing to a tool, test it against your actual hardest shot, not a hero demo. Useful decision criteria:
- Reference capacity: how many images can be supplied at once, and are they weighted individually?
- Identity scope: is identity applied per frame, per shot, or only at the first frame?
- Keyframe control: can you start from a locked still and control camera movement?
- Temporal stability: does detail boil or shimmer between frames?
- Clip length and resolution: enough for your edit, without aggressive upscaling artifacts?
- Iteration speed: how long does one re-roll take at your working resolution?
- Batch or API access: can you process twenty shots without clicking twenty times?
- Character reuse: can the same identity be recalled in a new project without retraining from scratch?
- Handoff: what format do you get, and does it drop cleanly into your editor?
- Commercial terms: does your license cover the distribution you plan?
Cloud tools win on speed and convenience; local pipelines win on reproducibility and privacy. Many teams run both: cloud for exploration, a local or reproducible setup for the shots that must match exactly.
Failure modes and how to fix them
| Symptom | Likely cause | Fix |
|---|---|---|
| Face changes on camera turns | Reference set lacks side angles | Add three-quarter and profile references; reduce camera speed |
| Wardrobe shifts color or cut | Synonymous or reordered wardrobe tokens | Freeze the wardrobe block verbatim; add a wardrobe reference image |
| Character looks frozen | Identity strength too high | Lower reference weight; allow an expression reference |
| Hands warp during gestures | Motion too complex for clip length | Shorten the clip; keep hands out of frame or slow the action |
| Background flickers | Scene tokens conflicting with style tokens | Simplify the scene line; lock a background plate |
| Style bleed across shots | Style tokens mixed into identity block | Separate blocks; change style only where intended |
| Detail boils frame to frame | Insufficient temporal conditioning | Use a model with stronger temporal coherence; reduce motion |
Keeping a whole series consistent
Once a single scene works, consistency becomes a documentation problem. Create a style bible that holds each character sheet, the exact identity text block, approved reference images, wardrobe layers per episode, and the seed values that produced your best takes. Name files with a convention that includes character, angle, and version so you never accidentally reference an outdated face.
For multi-episode work, freeze the look before production: choose a color pipeline, decide on grain and lens character, and record it. A locked grade makes small identity imperfections far less visible. Audio matters too: a consistent voice and room tone makes an audience forgive micro-drift in the image, while a mismatched voice makes a flawless face feel wrong.
Build a small library of reusable shots, such as walking, sitting down, and turning to camera, all generated from the same identity. You can cut these into future scenes and save both time and consistency risk.
Quality control checklist
Run this before a shot leaves your desk:
- Compare the first, middle, and last frame against the character sheet side by side
- Check eyes, teeth, hairline, and any distinguishing marks at full crop
- Verify wardrobe color, layer order, and accessories against continuity notes
- Watch at full speed for rhythm, then at quarter speed for artifacts
- Confirm background elements stay stable across the cut
- Confirm any on-screen text, logos, or signage is spelled correctly and unbroken
- Check lip sync if the character speaks, and check that mouth shapes match the language
- Confirm the clip starts and ends on frames your editor can cut cleanly
- Archive the prompt, seed, references, and settings with the final file
FAQ
How many reference images do I actually need?
Three well-chosen references, a front, a three-quarter, and a profile, will outperform ten random photos. Add a full-body and an expression reference when your shot list includes wide shots or emotional beats.
Can I get consistent results from one photo?
Sometimes, for short, front-facing shots. The moment the camera turns or the character needs a new angle, a single reference becomes a guess. Generate additional angles from the trusted photo, review them critically, and use the corrected set.
Do I need to train a custom model?
Not for most projects. Multi-image fusion with a careful reference set and a locked prompt block covers episodic and short-form work. Training becomes worthwhile when you need the exact same character across dozens of projects or many lighting conditions.
Why does lowering identity strength sometimes improve results?
Because expression needs room. If identity conditioning is too aggressive, the model cannot deform the face for a smile, a shout, or a turn, and the result looks like a mask. Trade a small amount of similarity for believable performance, then anchor it with a reference image that already shows the expression.
Can I fix drift in post instead of re-generating?
Yes, within limits. Stabilization, color matching, subtle warp-based face alignment, and quick compositing can rescue minor drift. Structural changes to bone structure or wardrobe are usually cheaper to re-generate than to repair.
How long should each generated shot be?
Two to five seconds per action is the sweet spot for most tools. Longer clips accumulate drift because the model keeps re-sampling detail. Shoot more short clips and cut them together rather than fighting one long take.
What is the biggest mistake beginners make?
Rewriting the identity description between sessions. Even small wording changes shift the sampled face. Keep one canonical identity block and paste it everywhere, exactly as written.
Key takeaways
- Multi-image fusion replaces single-reference guessing with a fused identity built from several angles of the same character.
- Reference set quality beats quantity: clean, neutral, mutually consistent images with varied angles.
- Lock keyframes as stills first, then animate. Iterating on stills is faster and cheaper than fixing motion.
- Keep identity tokens frozen and verbatim; change only scene, camera, and motion blocks between shots.
- Diagnose failures before re-rolling, and record which variable you changed.
- Document everything in a style bible so a character survives not just the next shot, but the next episode.
Consistency is not a single button. It is a workflow: a disciplined reference set, a frozen identity block, keyframe-first generation, and a QA pass that catches drift before your audience does. Build that habit and multi-image fusion stops feeling like magic and starts behaving like craft.
