Why Character Drift Happens in the First Place
Ask anyone who has produced more than a handful of AI video clips what frustrated them most, and the answer is rarely resolution, motion quality, or render speed. It is the moment in the edit where the same character appears in three consecutive shots and looks like three different people. The jaw widens in shot two, the eyes change shape in shot three, and by shot five the audience has quietly stopped believing the story.
Character drift is not a bug in one specific tool. It is a structural consequence of how image-to-video and text-to-video models represent people. A face is not stored anywhere inside the model as a face. It exists as a region in a high-dimensional feature space, and every generation is a fresh sample from that region. When you change the prompt, the camera angle, or the lighting description, you nudge the sampling point. If nothing anchors the identity region, the model happily samples a nearby point that satisfies your words — a woman in a red coat — rather than your character specifically.
Three practical causes account for most visible drift:
- Text contradicts the reference. Your reference image shows a soft jawline and a wide-set gaze, but the prompt says sharp features and piercing eyes. The model averages the two and produces a third face.
- Identity is underspecified in words. Phrases like a young woman with dark hair describe millions of faces. The model fills the gap with whatever is most probable in its training distribution.
- Scene information bleeds into identity. Costume, location, and lighting tokens sit in the same prompt as the face description. Change the location and the face changes with it, because the model never learned to keep those concepts fully separate.
There is also a purely mechanical layer: if the face occupies only 120 pixels in the reference, there is simply not enough signal to anchor anyone. Motion blur, heavy beauty filters, and sunglasses destroy the same signal before generation even begins.
Understanding this matters because it tells you where to intervene. You are not trying to describe a person more poetically. You are trying to give the model two independent measurements of the same identity and keep every other variable as boring and repeatable as possible.
What Two Reference Images Can and Cannot Do
Two references work like a two-point measurement. A single image pins down one view of a face, which the model can copy but cannot triangulate. A second image — ideally from a different angle or under different light — gives the model a second data point. From two points, geometry becomes inferable, and more importantly, responsibility can be split: one image carries identity, the other carries the look of the scene.
That split is the real trick. If both references try to define the face, they compete. If one defines who the character is and the other defines where the character is, they cooperate.
What two references realistically support:
- Front-facing, three-quarter, and mild profile views of the same character
- Moderate expression changes — smiling, speaking, listening
- Consistent wardrobe and hair within a scene or a short sequence
- Stable identity across five to fifteen consecutive shots when you feed approved frames forward
What they will not reliably give you:
- A true full rotation around the head
- Accurate back-of-head and extreme profile views
- Consistent hands during fast action or object handling
- Fine detail under extreme close-up, such as teeth or individual eyelashes
- Reliable behaviour when the face is turned more than roughly 45 degrees from the reference
The practical conclusion is not to fight those limits but to design around them. Directors have done this forever: when a stunt double cannot be shown clearly, you cut to the reaction, the silhouette, or the over-the-shoulder framing. A two-reference workflow rewards the same instinct. If a shot demands a hard profile or a fast spin, plan a cutaway instead of gambling an entire sequence on a view the model has no data for.
Choosing the Reference Pair: The 70/30 Rule
Not all reference pairs are equal. A useful heuristic: roughly 70 percent of your consistency comes from one clean identity image, and 30 percent from one context-matched image. If you only have time to improve one thing, improve the first.
The identity image
This image is your character's passport photo. Its only job is to be unambiguous.
- Neutral expression, mouth closed, eyes open and looking toward the lens
- Flat, soft, even lighting with no harsh shadow crossing the face
- Nothing occluding the face: no hat brim, no sunglasses, no hand, no hair strands over the eyes
- Sharp focus, at least 1024 pixels on the short side, matching the aspect ratio of the video you plan to make
- Shoulders and upper chest visible, so the model also learns neck length and body proportions
The context image
The second image is where the scene lives. Same person, but now wearing the costume, standing in a similar lighting mood, ideally at a three-quarter angle.
- If your scene is night neon, use a reference lit by something warm and directional, not by flat daylight
- Keep the face readable. A moody reference with the face in shadow teaches the model to hide the face
- Match the lens feel if you can. A wide-angle reference paired with a long-lens prompt creates subtle distortion mismatches
Pairs to avoid
Two near-identical frames add almost no information; the second image does nothing. Two images with different hairstyles, ages, or body weights are worse than useless, because the model averages them into a stranger. Heavily retouched images strip out skin texture, freckles, and asymmetry — exactly the small signals that make a face feel like a specific person.
Run a two-second sanity check: place the pair side by side and squint. If a stranger could not confidently say these are the same person, the model will not either.
The Identity Block: Building the Prompt That Holds a Face
Most drift is a prompt architecture problem. Instead of writing a fresh sentence for every shot, build a fixed block and change only the variables around it.
A workable structure looks like this:
- Identity line — the same words, character for character, in every shot
- Wardrobe line — the same words until the story changes clothes
- Action line — what happens in this specific shot
- Camera line — framing, lens feel, height, and movement
- Lighting line — direction, quality, and colour temperature
- Style line — the overall look, kept identical across the sequence
A realistic identity line might read: ID: Mira, late twenties, oval face with a soft jaw, wide dark eyes, thick straight black hair to the collarbone, small mole under the left eye, neutral expression.
An action line for shot three might read: ACTION: she turns from the window and lifts a ceramic cup with both hands.
The discipline is simple and slightly tedious: never redecorate the identity line. Do not add relaxed smile to the identity line for a happy shot and serious mood for a sad one. Put emotional state in the action line, where it belongs. If you want the character to smile, say so in the action, not in the identity.
Keep a short negative list as well — face morph, age change, facial reshaping, warping, extra fingers, duplicated features. Negative prompts are blunt instruments, but for identity work they reliably suppress the most common failure modes.
Finally, decide on a seed policy. If your tool supports seeds, reuse the same seed across shots in a scene. A fixed seed narrows the sampling space and gives the identity anchor less room to wander.
Step-by-Step Workflow: From Two Images to a Consistent Sequence
This is the process that scales from a fifteen-second test to a multi-scene narrative short.
1. Write the shot list before you generate anything. Columns for character, wardrobe, time of day, location, framing, and motion. You cannot maintain continuity you never defined. A ten-line table saves hours of regeneration.
2. Assemble the reference folder. One identity image, one context image, plus any approved frames from previous shots. Keep it tidy and versioned.
3. Generate stills before video. Image generation is far cheaper than video generation in both time and cost. For a difficult shot, produce twenty to forty still candidates and pick one. Only winners get animated.
4. Approve one keyframe per shot. Name it clearly, for example scene01_shot03_approved. Approved frames become the visual contract for everything downstream.
5. Animate from the approved keyframe. Feed the keyframe as the starting frame, paste the identity block unchanged, and match the seed if your tool allows it.
6. Feed forward. Once a shot is good, export its strongest frame and add it as a third reference for the next shot. This rolling reference technique is where a two-image workflow turns into genuine long-sequence continuity. Ten shots later, your reference set encodes the character as they actually appear in the film, not just as they appeared in the original photograph.
7. Check the joins. Place the last frame of shot N beside the first frame of shot N+1 and compare at full size. A join is where audiences notice drift most, because the eye compares the two images directly.
8. Assemble, then fix locally. When you review the cut, resist the urge to regenerate entire sequences. Isolate the two or three shots that drifted and repair those.
Shot-Type Playbook: Where Identity Breaks and How to Shoot Around It
Close-ups and dialogue
Close-ups are the easiest shots for identity and the most unforgiving for errors, because the face fills the frame. Keep motion subtle — a slow push-in or a small head turn. Avoid fast expression changes in a single clip; split a smile into two shots rather than asking one clip to travel from neutral to laughing.
Wide and full-body shots
Here the face occupies a small fraction of the image, so identity signal per pixel collapses. The audience stops scrutinising the face and starts reading silhouette, wardrobe, and posture — which gives you latitude. Anchor the character mid-frame, keep costume colours consistent, and return to a close-up within a shot or two so the eye re-confirms who this is.
Profile and back-of-head
This is the weak spot. Use a reference with that exact angle if one exists. Otherwise, prefer over-the-shoulder framing, silhouettes, or a cutaway to a listener. Never build an emotional beat that depends on a hard profile the model has no data for.
Action and fast motion
Motion blur erodes facial detail faster than any other factor. Shorten action clips to two to four seconds, reduce pan and tracking speed, and make sure the face is visible at least part of the time. If a scene needs a long chase, cut it into short beats rather than one long continuous move.
Multiple characters in one frame
Give each character its own reference pair and describe positions explicitly — left of frame, mid-ground, seated at the table. Generate each character alone first to confirm their identity holds, then combine. Do not expect a single clip to establish two new faces at once.
Continuity Beyond the Face: Wardrobe, Lighting, and Props
Audiences forgive a slightly different nose far more readily than they forgive a jacket that changes colour between cuts. Protect the boring details.
Wardrobe. Choose a precise costume sentence and reuse it verbatim. Dark blue is ambiguous; navy wool coat with three visible buttons is not. Wardrobe is also the first thing to break when you edit prompts in a hurry.
Lighting. State direction and quality explicitly — soft window light from camera left, warm practical lamp behind the subject. Flipping light direction between shots reads as a continuity error even when the face is identical, because the shadows on the face change shape.
Props and hair. Track which hand holds what. Hair length and texture changes make a character read as a different person faster than almost any facial variation, so keep hair description in the identity block and never touch it mid-sequence.
Environment. If a scene changes location, change only the environment line. Everything above it stays frozen.
Troubleshooting: Symptom, Cause, Fix
- The face becomes a generic person. The identity line is too thin, or the two references disagree. Tighten the identity description and replace the weaker reference with a cleaner front-facing image.
- The character ages between shots. References come from different life stages, or the prompt contains age words that conflict. Use same-age references and remove adjectives like young or mature from the action lines.
- The style shifts — live action turns painterly, or 3D turns illustrated. Style tokens differ between shots, or two style phrases are fighting. Freeze one style line and reuse it everywhere.
- The face holds but the clothes change. Wardrobe is embedded in the identity line. Split them into separate lines so edits to one do not disturb the other.
- Warping around the mouth and eyes. The keyframe resolution is too low or the motion strength too high. Raise keyframe resolution and reduce motion.
- Hands flicker or merge with props. Hands are the least reliable region in most models. Reframe to keep hands out of shot, occlude them, or slow the action down significantly.
- Identity holds in stills but collapses in motion. The animated clip is too long. Cut the clip length and generate two shorter shots instead.
Quality Control and Common Mistakes
Review at low resolution first. Build a contact sheet of candidate frames, approve in two separate passes, and only then spend time on higher-quality output. The first pass asks one question: does this read as the same person? The second asks: does this match the shots on either side of it? Mixing the two questions in one pass makes you accept a shot that is faithful but discontinuous.
Keep a locked folder of approved frames, rate takes simply — keep, maybe, discard — and delete the discards immediately so you are never tempted to revisit them.
The mistakes that break consistency most often:
- Using two references shot in the same pose, so the second image adds nothing
- Rewriting the identity sentence between shots for variety
- Judging identity at thumbnail size instead of full frame
- Regenerating an entire sequence when a single shot drifted
- Leaving motion strength at a default that produces dramatic but blurry action
- Starting generation without a shot list, leaving nothing concrete to be consistent with
- Trying to solve story problems and visual continuity problems in the same pass
FAQ
Can two reference images really hold a character together for a whole video?
For a single scene and shots within the reference range, yes. For longer projects, treat the original pair as a starting point and keep adding approved frames as the sequence grows. The effective reference set becomes a rolling record of the character as they appear on screen.
Is more references always better?
No. References that contradict each other are worse than a single strong image, because the model averages them into a face that belongs to nobody. Two clean, non-conflicting references beat six mediocre ones. Add a third only when it is an approved frame from the same sequence.
Should the references be photographs or AI-generated images?
Both work. AI-generated references are easier to control for lighting and wardrobe, but they arrive with their own small artifacts, which the video model will faithfully reproduce. Photographs carry more real facial texture but may need cropping and colour correction before use.
What resolution do the references need?
The face should be at least 300 to 400 pixels wide, and the file should be at least 1024 pixels on the short side. Match the aspect ratio of your target video to avoid unwanted reframing.
Why does the face change when the background changes?
Scene and identity information are not perfectly separated in most models, so costume color, location, and lighting tokens leak into the face. Keeping a frozen identity block and changing only environment lines reduces this considerably.
How long should each clip be?
Two to five seconds for identity-critical dialogue or close-ups. Wide shots tolerate longer clips because facial detail matters less and the audience has less to compare.
Putting It Into Practice
Consistency is not a feature you switch on; it is a workflow you maintain. Two well-chosen reference images give you the raw material — one identity anchor and one scene anchor. A frozen identity block protects that material from your own prompt edits. A shot list gives the sequence something to be consistent with. Rolling approved frames forward turns a short test into a sustainable production pipeline.
Start small. Pick one character, one scene, and five shots. Build the pair, write the identity block, generate stills, approve keyframes, animate, and compare your joins. If all five shots hold, you have a process you can repeat for an entire project. If they do not, the troubleshooting list will tell you exactly which of the four variables — references, identity block, motion, or continuity data — needs attention before you generate another clip.
