Why characters drift between shots
Anyone who has tried to build a short film, a serialized ad, or an episodic explainer with AI video hits the same wall. The first shot looks great. The second shot looks great too — except the face is slightly wider, the hair has changed color, the jacket lost its collar, and the eyes sit a little too far apart. By shot six, the audience is no longer watching a character. They are watching a stranger in a similar outfit.
This is not a rendering bug. It is the natural consequence of how diffusion-based video models work. Each generation starts from noise and is steered by a prompt plus whatever conditioning signals you supply. If the only signal is text — "a woman in her thirties with auburn hair, denim jacket, soft daylight" — the model is free to sample any plausible woman who matches that description. Every sample is a new interpretation of the same words.
Text-to-image models have the same problem, but video makes it worse for three reasons:
- Temporal exposure. A still image gives you one chance to notice a mismatch. A moving shot holds the face on screen for seconds, and small inconsistencies become obvious.
- Camera movement. As the camera pans or the subject turns, the model must invent the profile, the back of the head, and the ears. Those regions were never well specified in the prompt.
- Multi-shot editing. A story is told through cuts. Each cut is often a separate generation, which means a separate identity sample.
Multi-image fusion is the technique that solves this. Instead of describing a character in words, you hand the model several images of the same character and let it fuse them into a single, stable identity that can be reused across shots, angles, and lighting conditions.
This guide covers what happens inside that fusion process, how to build reference material that actually works, how to write prompts that preserve identity, a step-by-step workflow you can reuse, and the failure modes that waste the most time.
What multi-image fusion is actually doing
At a high level, multi-image fusion converts several reference images into a compact numerical description of a character, then injects that description into every generation. It sounds simple. In practice there are three distinct sub-problems, and understanding them changes how you prepare your references.
Identity extraction versus style extraction
A good fusion pipeline separates who the character is from how the scene looks. Identity lives in stable geometry: face shape, interpupillary distance, nose bridge, jawline, hairline, skin tone. Style lives in everything else: the color grade, the lens, the film grain, the background, the wardrobe styling.
If you feed the model five images that all share a heavy teal-orange grade, it may treat that grade as part of the character. Your next shot, lit with neutral daylight, will fight the identity embedding and produce a washed-out, confused result. The fix is to supply reference images that vary in lighting and background while keeping the face consistent. That variance teaches the model which features are invariant.
The embedding space problem
Models do not compare faces pixel by pixel. They map images into a high-dimensional embedding space where similar faces cluster together. When you supply multiple references, the system typically averages, clusters, or otherwise reduces them to a representative point — sometimes several points if the model supports multiple identity tokens.
Two practical consequences follow:
- Outliers poison the average. One reference with an unusual expression or a strong shadow pulls the identity point away from the true center. Five clean references beat eight mixed ones.
- Consistency within the reference set matters more than realism. Slightly soft, evenly lit, front-facing images often outperform cinematic portraits, because they map more tightly.
Continuity verification and keyframe control
Once identity is injected, the model still has to decide how the character moves. Many pipelines use a keyframe strategy: generate or select anchor frames at the start and end of a shot, then let the model interpolate motion between them. Because both anchors carry the same identity conditioning, the interpolated frames inherit it.
Verification is the unglamorous half of this. Before committing to a long render, generate a low-resolution sweep across the shot's key poses — front, three-quarter, profile, back — and check whether the face holds. This costs a fraction of a full render and catches most identity drift early.
Building a character reference pack that works
A reference pack is the small library of images you reuse for a character across an entire project. Treat it as a deliverable, not an afterthought.
How many images, and which angles
For most modern pipelines, five to ten references is the sweet spot. Fewer than four and the model has too little signal. More than fifteen and you start averaging in noise, plus contradictory lighting.
A reliable starter pack:
- One neutral front-facing portrait, even lighting, no strong shadows
- One three-quarter view, left
- One three-quarter view, right
- One true profile
- One slight upward angle and one slight downward angle
- Two or three images in different scenes or outfits, same face
That last category is the one most people skip, and it is the one that teaches the model what is invariant.
Technical quality beats artistic quality
Reference images should be sharp, well exposed, and free of heavy filters. Practical guidelines:
- Resolution: at least 1024 pixels on the short edge. Upscaled, blurry references produce blurry identities.
- Framing: head and shoulders, with some margin. Cropped chins and half-faces confuse feature extraction.
- Expression: neutral or a mild smile. Extreme expressions distort the geometry the model is trying to learn.
- Background: simple and uncluttered. Busy backgrounds leak visual style into the identity.
- Accessories: decide early whether glasses, hats, or masks are part of the character. If they appear in some references but not others, the model will hallucinate them intermittently.
What to leave out
Exclude anything that contradicts itself across the set: different hair colors, drastically different makeup, heavy motion blur, watermarks, text overlays, or images where the face occupies less than a fifth of the frame. Also exclude group photos unless the character is unambiguously the subject — a fusion model asked to average three faces will produce a person who resembles none of them.
Prompting so identity survives the shot
Multi-image fusion reduces identity drift, but it does not eliminate it. Prompts still steer the generation, and vague prompts give the model room to reinterpret the face.
Anchor the invariants in text too
Even with strong references, restate the two or three features that matter most: hair length and color, eye color, and one distinctive marker such as a scar, a mole, or a consistent accessory. Redundancy across modalities — image plus text — measurably improves stability.
Describe motion, not appearance
Prompts that re-describe appearance compete with the reference images. Prompts that describe action, camera, and environment cooperate with them.
| Weak prompt | Strong prompt |
|---|---|
| A woman with auburn hair and green eyes standing in a café | She lifts the cup, steam rising; slow push-in, warm morning light through the window |
| A man in a grey coat walking | He walks away from camera down a wet street, handheld follow, shallow depth of field |
Lock the constants across the project
Create a character prompt block — a short, fixed paragraph describing the character — and paste it at the top of every shot prompt. Then append shot-specific instructions. This turns a creative decision into a template, which is exactly what consistency requires.
A practical template:
CHARACTER: [name], [age range], [hair], [eyes], [distinctive feature], [default wardrobe]
SHOT: [action], [camera move], [framing], [environment], [lighting], [mood]
NEGATIVE: [unwanted traits]
A repeatable shot-by-shot workflow
This is the sequence that produces the fewest wasted renders across most projects.
Step 1 — Write the beat sheet first. Break the scene into shot functions: establishing, reaction, action, transition. Identity problems are most visible in reaction shots, so plan for them rather than discovering them.
Step 2 — Build the reference pack before generating any shot. Shoot or generate the reference images and review them as a set. If one image looks different from the others, remove it now, not later.
Step 3 — Generate identity tests at low resolution. Produce four still frames per shot idea: front, three-quarter, profile, and one extreme angle. Review them side by side in a contact-sheet layout. This is the cheapest quality control step in the whole pipeline.
Step 4 — Approve anchors, then render motion. For each shot, pick the first and last frame from your approved stills. Animate between them. Motion models drift far less when both endpoints are locked.
Step 5 — Keep a continuity log. A simple spreadsheet with columns for shot number, wardrobe, hair state, time of day, and which reference pack version was used. When shot 14 looks wrong, the log tells you whether the problem is the prompt or a superseded reference set.
Step 6 — Batch similar shots together. Generate all the dialogue close-ups in one session, all the wide shots in another. Switching repeatedly between a close-up identity and a distant-silhouette identity increases drift and makes troubleshooting harder.
Step 7 — Do a full-sequence review at draft quality. Watch the assembled cut before polishing any individual shot. Continuity issues that are invisible in isolation become obvious in sequence, and fixing them at the assembly stage is much cheaper than re-rendering final shots.
Choosing the right tool for each stage
Most teams end up with a small stack rather than a single application. When evaluating options, compare them on these axes instead of on feature lists.
- Reference capacity. How many images can be conditioned at once, and does the tool weight them or average them? Tools that accept a primary reference plus supporting images usually hold identity better.
- Identity reuse across sessions. Can you save a character profile and return to it next week with identical results? Session-persistent identity is worth more than a marginally better single render.
- Motion control granularity. Keyframe interpolation, camera path control, and motion strength adjustment all reduce drift. A tool with weak motion control forces you to fight the model.
- Aspect ratio and resolution flexibility. If you need vertical, square, and widescreen cuts of the same scene, confirm the identity survives a re-frame.
- Export and handoff. Clean frame sequences, alpha support, and predictable file naming save hours in post.
- Iteration speed at draft quality. Fast, cheap previews matter more than peak quality, because consistency is a volume game.
A common and effective split: one image model for building the reference pack and anchor frames, one video model for motion, and one compositing tool for fixes. Keeping the reference pack in a plain folder with versioned names — character-name-v3-front.png — means you can swap models without rebuilding your character from scratch.
Failure modes and how to fix them
The face morphs mid-shot
Usually caused by a single anchor frame or by motion strength set too high. Fix: lock both endpoints, reduce motion strength, and shorten the shot. Long continuous takes are far harder to keep stable than two or three shorter cuts.
The character looks right but the age keeps shifting
Age is one of the least stable attributes because it is encoded subtly in skin texture and facial volume. Add explicit age language to the prompt and include at least two references with visible skin detail. Avoid references that have been heavily retouched.
Wardrobe changes color between shots
Color drifts when the color grade of the reference images differs from the target scene. Fix by specifying wardrobe color in text as well as image, and by removing references whose grade fights the intended look.
The identity is correct but the lighting is wrong
This is the opposite problem and usually means the reference set was too uniform in lighting, so the model baked your studio setup into the identity. Add references shot or rendered under different lighting to decouple the two.
Backgrounds leak between shots
If your reference images all share a distinctive location, the model may reproduce it. Use references on neutral backgrounds, or generate your reference pack specifically for the purpose rather than pulling frames from finished scenes.
Two characters in one frame merge
Multi-character scenes are the hardest case. Generate each character separately first, verify both hold identity, then compose. Some tools support multiple identity tokens; if yours does not, use over-the-shoulder framing, split-screen cuts, or staging that keeps faces apart.
Continuity at scale
Once a character survives a single scene, the challenge shifts to sustaining them across episodes, campaigns, or a whole channel. Three habits make the difference.
Version your reference packs. A character is not static. Hair grows, wardrobe changes with the season, and a scar may heal. Keep v1, v2 folders and note in the continuity log which version each shot used. When you return months later, the log is the only thing standing between you and a reshoot.
Write a character bible. Half a page: name, age, physical description, wardrobe defaults, three personality traits, and the reference pack version. Anyone joining the project can then produce a consistent shot without asking you.
Standardize the shot template. If every episode uses the same prompt structure and the same aspect ratio, the model's behavior becomes predictable. Predictability is the real goal — consistency is just its visible result.
Frequently asked questions
How many reference images do I really need?
Five to eight well-chosen images cover most characters. If your character appears only in wide shots, three or four may be enough. If they appear in close-up dialogue, aim for eight to ten with varied angles.
Can I use an AI-generated image as a reference?
Yes, and many teams do. Be aware that generated references inherit the quirks of the model that made them. Generate the pack once, review it as a set, and then treat it as fixed source material.
Why does the character look perfect in stills but drift in video?
Because motion adds frames the model must invent. Identity conditioning is strongest when the pose resembles the references. Turning, occluding, or moving toward the camera all push the model into territory your references did not cover. Add angle variety to the pack and lock keyframes.
Does multi-image fusion work with real actors?
It can, provided you have the rights to the footage and a clear consent agreement. Using a real person's likeness raises legal and ethical questions that are outside the technical scope, so document permissions before you start.
How do I fix a character that has already drifted in earlier shots?
Do not patch every shot. Rebuild the reference pack, re-lock keyframes, and re-render only the drifted shots at draft quality first. Once the sequence holds at draft quality, push the approved shots to final.
Is it better to create one long take or many cuts?
Many cuts. Each cut resets the generation, which is a small risk, but it also limits how far drift can accumulate. Long takes compound small errors into large ones.
What single change improves consistency the most?
Locking both the first and last frame of every shot. It constrains the interpolation and removes most of the freedom the model would otherwise use to reinterpret the face.
Do I need a different workflow for stylized or animated characters?
The principles hold, but stylized characters are more forgiving of small geometric differences and less forgiving of line work and shading changes. Keep line weight and shading style consistent across the reference pack, and add style descriptors to your fixed character block.
