Why character drift ruins otherwise strong AI video
You can have the best lighting, the sharpest motion, and the most ambitious camera move in the world, and a single morphing face will still make the whole clip feel amateur. Character drift is the quiet failure mode of generative video: the model renders a convincing human in shot one and a slightly different human in shot two. The cheekbones soften, the jaw widens, the eyes shift from hazel to brown, the hairline recedes by a centimeter. Individually, each frame looks fine. Played in sequence, the illusion collapses.
Audiences are extremely good at this. Human perception is tuned for faces, and we notice identity swaps long before we notice a mismatched shadow or a floating prop. In narrative work, that sensitivity is fatal. If a viewer reads two different people in a two-shot conversation, the story stops being a story and becomes a slideshow of unrelated images. In commercial work, the damage is more direct: a brand ambassador who changes appearance between cuts reads as inauthentic, and in a product video that undercuts the exact trust the spot was built to create.
The cost shows up downstream. Teams burn hours regenerating shots, then burn more hours in editing trying to hide the seams with cuts, reframes, and motion blur. Sometimes the fix is to abandon a promising scene entirely. The root cause is rarely the video model itself. It is almost always the way the character was described to the model in the first place.
Multi-image fusion is the technique that addresses that root cause. Instead of asking a model to invent a person from a text description, you hand it several images of the same person and let it derive a single, stable identity that it can reuse across every shot. Done well, it turns a pile of loosely related clips into a continuous performance.
How multi-image fusion builds a stable character identity
From reference frames to identity embedding
A fusion pipeline rarely looks like magic from the outside. It is a sequence of unglamorous steps. Each reference image is passed through a vision encoder that converts pixels into a numerical description of the face: the distance between the eyes, the slope of the nose bridge, the width of the jaw, skin undertone, hair silhouette, brow shape, and dozens of other measurements. Those descriptors are then combined into a single identity representation, sometimes called an embedding or an identity token set, that the video generator can condition on.
The reason multiple images beat one image is statistical. A single photo bakes in the pose, the lighting, and the expression of that one moment. The model cannot tell which features belong to the person and which belong to the situation. Give it six photos taken at different angles under different light and it starts to separate the constant from the variable. The chin shape stays because it appears in every frame; the harsh side shadow disappears because it does not. Fusion is essentially a noise-averaging operation applied to identity.
Weighting matters too. Most implementations let you decide how strongly each reference image influences the final result. A clean, evenly lit three-quarter shot usually deserves more weight than a dramatic profile with half the face in darkness. If your tool does not expose weighting directly, you can approximate it by simply removing weak references from the set. Fewer, better images consistently outperform a large, messy pile.
What the model locks and what it reinterprets
It helps to think of consistency as a constraint problem rather than a memory problem. The model is not remembering your character; it is being constrained by an identity representation while it solves a new problem, which is generating motion that looks plausible. Everything that does not conflict with the identity constraint gets reinvented for each shot.
Locked, in practice: overall facial geometry, apparent age, skin tone, eye color, hair color and general length, and any accessory that appears in most references. Reinterpreted, in practice: exact hair strand placement, fabric folds, how clothing drapes, ambient light color, background detail, and micro-expression timing. That is why a jacket can look slightly different in every scene even when the face is perfect. Knowing the boundary tells you where to spend your effort. Fight for the face. Accept minor variation in everything else, or control it with separate tools.
Designing a reference set that survives every shot
The reference set is the single highest-leverage asset in the whole pipeline. Teams that spend twenty minutes curating images save hours of regeneration later.
Angle coverage
Aim for five to eight images. The essential ones are: a straight-on neutral shot, a three-quarter view from the left, a three-quarter view from the right, a near-profile, and one shot from slightly above or below eye level. Add a smiling frame if the performance requires warmth, because a model conditioned only on neutral expressions tends to render a smiling version of the character as a different person.
Avoid extreme angles. A full profile or a dramatic worm's-eye view teaches the model very little about the face and a lot about distortion. If you only have wild angles available, generate or capture a few clean reference shots first, then use those.
Light and lens discipline
Keep the reference lighting soft, even, and frontal. Hard shadows create high-contrast regions that the encoder may interpret as facial structure. Mixed color temperature is worse: a warm kitchen light in one reference and a cool window in another can push skin tone estimates in two directions at once.
Lens choice matters as well. Wide-angle portraits stretch features near the edges of the frame; long lenses compress them. Pick one look and stay with it. If your references were shot on different focal lengths, the model averages a face that does not quite match any of them. Also skip beauty filters, heavy grain, and stylized color grades on reference images. Save the look for the final render.
Wardrobe and continuity anchors
Decide which outfit is the hero look and provide at least three images of it. If the character appears in a second outfit, build a small secondary reference set and treat it as a separate conditioning run rather than mixing both wardrobes into one set. Mixing is the fastest way to get a character who wears a strange hybrid of two costumes.
Props are powerful anchors. A specific pair of glasses, a scarf, a scar, a distinctive necklace, or a signature hairstyle gives the model an additional constant to hold onto. Just be consistent about it. A prop that appears in half the shots and vanishes in the other half creates ambiguity about what the character actually looks like.
A simple rule for the whole set: everything that should never change must appear in most of the references, and everything that should change must appear in as few as possible.
A practical fusion workflow for a multi-scene sequence
This is the part most guides skip. Fusion is not a button; it is a loop with a start and a stop condition.
Step 1: turn the script into a beat sheet
Before generating anything, break the script into beats. A beat is a unit of meaning, not a shot. Introduce character, reveal intent, escalate conflict, resolve. Write each beat in one sentence, then note the emotional register. Consistency problems often start here, because a beat sheet reveals that a character has to look tired in act two and rested in act one, and that difference must be planned rather than discovered in the render.
Step 2: build the shot list and the still library
Convert each beat into one to four shots, each with a camera description, a duration, and a lighting note. Then produce a still frame for every shot before generating any motion. Stills are cheap. Video is not. Reviewing a storyboard of generated stills catches identity problems while they are still trivial to fix.
When generating stills, keep the prompt lean. Describe the scene, the action, the light, and the framing. Do not re-describe the character in detail. The identity representation is already carrying that load, and redundant text descriptions fight with the images. If you must add character text, use only traits that are not visible in the references.
Step 3: generate in passes, then compare
Generate a short test clip for the character's first appearance, then a second clip for their next appearance, and compare the two side by side at the same scale. Do not compare a wide shot against a close-up; you will fool yourself. Crop both to the face and place them next to each other. If the jaw, hairline, or eye spacing shifts visibly, stop and fix the references before generating anything else.
Once the pair holds up, expand in small batches. Generating forty clips at once and then discovering a systematic drift is the most expensive mistake in this workflow.
Step 4: assemble, patch, and lock
Cut the clips together early, even at low quality. Sequence exposes problems that isolated clips hide, because the eye compares adjacent frames in a way that a thumbnail grid does not. When a shot fails, patch it locally: regenerate just that clip with a tighter reference weighting, use a still frame as the first frame, or trim the shot so the drift happens off screen.
When a sequence finally holds, freeze it. Save the reference set, the weighting values, the prompts, and the model settings as a named character preset. The next episode or the next campaign should start from that preset, not from scratch.
Decision criteria: when fusion is enough, when to change tactics
Not every project needs the same level of investment. Use these criteria to decide how far to go.
- Single shot, no repeat appearances: skip fusion. Use a text prompt and accept variation. It will not matter.
- Two to five shots, photoreal, same character: fusion is enough, plus a tight reference set and a comparison pass.
- Series or episodic content: build a reusable character preset and a written style guide, then treat the preset as a production asset with version numbers.
- Talking-head or presenter content: lock the identity and also lock the framing and lighting. Presenter formats reward visual stability far more than cinematic variety.
- Action-heavy sequences: expect identity stress. Fast motion, extreme poses, and heavy occlusion push models into extrapolation. Shorten shots, favor wider framing during movement, and return to a close-up when the character is still.
- Stylized or animated looks: fusion still helps, but you may need stylized reference art rather than photographs. Mixing a photo reference with an illustrated target can produce an uncanny middle ground.
- Rapid style changes mid-project: lock identity first, then apply style. Inverting that order forces you to rebuild the identity every time the look changes.
One more criterion is worth stating plainly: if a character has a very unusual face, expect more work. Distinctive features are anchors, but they are also the features most likely to be exaggerated or smoothed away under motion. Test early with the hardest shot in the sequence, not the easiest.
Advanced controls: motion, physics, and style transfer
Once identity holds in static shots, motion becomes the next frontier. The character can look perfect in every frame and still feel wrong in sequence because the way they move is inconsistent.
Start with conservative camera language. Slow push-ins, gentle drifts, and locked-off frames give the model fewer chances to reinterpret the face. Save the whip pans and handheld energy for shots where the character is small in frame.
Use keyframes deliberately. Many pipelines let you supply a starting frame, an ending frame, or both. If a character must end a shot in a specific pose, supply a still of that pose. It reduces the amount of invention the model has to do and therefore reduces drift.
Style transfer should come after identity, never before. If you stylize the reference images first, you are teaching the model a painterly face that may not correspond to the actor or the character design. Lock the person, then apply the grade, grain, or illustration treatment as a finishing layer.
Physics and cloth are the last things to worry about, and often the easiest to hide. Sleeves that behave slightly differently between shots rarely break a scene. Hair that changes length does. Prioritize accordingly: face, hair silhouette, body proportions, color palette of clothing, then fabric detail.
Mistakes that cause identity drift and how to fix them
- Too many low-quality references. Fix: cut the set to five to eight clean, evenly lit images.
- References with different lighting setups. Fix: reshoot or regenerate references under one consistent light.
- Re-describing the character in every prompt. Fix: strip physical descriptions from prompts once the identity references are set.
- Mixing outfits in one reference set. Fix: separate presets per costume.
- Testing with easy shots. Fix: test the hardest shot first, such as a fast turn or a close-up with strong shadows.
- Judging consistency from thumbnails. Fix: crop faces and compare at full size.
- Generating long clips. Fix: keep shots short, three to six seconds, and cut rather than push the model to sustain a long take.
- Forgetting to save the settings. Fix: store prompts, weights, seeds, and model version in a preset file.
- Changing model versions mid-project. Fix: finish the sequence on one version, or re-run the comparison test after any update.
- Ignoring the background. Fix: an unstable background draws attention away from small face drifts, but a shifting background plus a shifting face is unusable, so stabilize what you can.
A tool-agnostic pipeline and handoff checklist
You can run this workflow with any generative video stack. What makes it survive a team is documentation.
Name files predictably: project_charactername_shotnumber_version. Keep a character folder containing the reference set, a plain-text note listing which references carry the most weight, and one approved hero shot. Store prompts next to the outputs they produced, not in a separate document that drifts out of sync.
Define a review gate. A shot is approved only when its face matches the hero shot at matched scale, its wardrobe matches the costume preset, and it cuts cleanly against the two shots on either side. Review in sequence, never in isolation.
Finally, record the failure modes you encountered and what fixed them. A two-page internal note is worth more than any tutorial, because it documents your specific model, your specific character, and your specific look.
FAQ: practical questions about consistent characters
How many reference images do I need?
Five to eight well-lit, varied-angle images is the sweet spot for most projects. More than twelve rarely improves results and often introduces conflicting lighting.
Can I use images of a real actor or a public figure?
Only with appropriate rights and permissions, and only where the intended use is lawful. Many tools also restrict generating recognizable real people. Prefer original character designs or fully licensed talent.
Do I need to re-describe the character in every prompt?
No. Once identity references are attached, extra facial description usually hurts. Describe action, setting, camera, and light instead.
Why does the face hold but the clothes change?
Clothing is not part of the identity representation in most pipelines. Lock it with a separate costume reference set or accept minor variation and stabilize it in post.
What about hands and body proportions?
These are separate failure modes. Keep hands small in frame where possible, or out of frame entirely. If proportions drift, generate a full-body reference and include it in the set.
How long should each shot be?
Three to six seconds is a practical range. Longer clips give the model more time to drift, and they are harder to patch.
Should I upscale before or after consistency checks?
After. Upscaling adds detail that can make small identity differences look more pronounced, so resolve consistency at native resolution first.
What if the model updates and my character changes?
Re-run the comparison test. Rebuild the preset if needed. This is why version documentation matters.
Is fusion worth it for a one-off social clip?
Usually not. For a single short clip, an image-to-video approach with one strong starting frame is faster and nearly as good.
Putting it into practice: a four-week consistency drill
Week one: build three reference sets for three different characters, each with six images, and document your selection reasoning. Week two: generate a five-shot sequence for one character and log every drift you spot. Week three: rebuild the same sequence with a corrected reference set and compare the two versions side by side. Week four: produce a thirty-second piece with two characters in the same scene and hold both identities.
By the end, you will have something more valuable than a tuned prompt: a repeatable process. Consistency in AI video is not a trick you discover once. It is a discipline of preparation, small tests, and ruthless comparison, and it is the difference between clips that happen to look alike and stories that hold together.
