Why AI video drifts, and what multi-image fusion changes
Imagine you are making a two-minute short about a lighthouse keeper named Elin who repairs a broken lamp before a storm. You write a prompt for shot one and get a beautiful frame: a woman in her fifties with a grey braid, an oil-stained canvas jacket, standing on a spiral staircase in cold blue light. Shot two asks for the same woman climbing the stairs. The result is a different person, softer jaw, brown hair, a clean coat, warm sunlight. Nothing matches. Multiply that by thirty shots and the film has fallen apart before you ever reach the edit.
That failure mode is called drift, and it is the single biggest reason AI video projects stall. Text prompts are lossy descriptions. Phrases like a weathered face or a cozy workshop leave the model enormous freedom, and the model exercises that freedom independently for every generation. Identity, wardrobe, props, location, lighting, and color grade all wander.
Multi-image fusion attacks the problem at its root. Instead of describing a character in words and hoping the model lands close enough, you hand the model several still images of that same character and let it build an internal profile, a compact representation of what makes this person look like this person. Every later shot is conditioned on that profile rather than on adjectives.
The practical shift is that your still images become the source of truth. Prompts narrow down to what actually changes: action, camera, mood, time of day. Continuity stops being a gamble and becomes a property of your reference library.
Three things improve at once. First, recognizable identity: faces, hair, and body proportions stay stable. Second, production speed: far fewer regenerations and far less manual repair. Third, storytelling: viewers accept a scene when their eyes are not constantly re-learning who is on screen. Consistency is not a cosmetic nicety. It is the difference between a clip and a story.
The core mechanism: how a profile is built from several stills
Multi-image fusion, sometimes described as reference-conditioned generation, is a pipeline rather than a single switch. Knowing its stages makes debugging dramatically easier.
Reference intake and normalization. Your images are cropped, resized, and sometimes face-aligned to a canonical pose so the encoder is not distracted by tilt or scale.
Subject isolation. A segmentation mask separates the character from the background. This matters because you often want the person from one set of images and the room from another. If you never separate them, the model may fuse the character with the studio backdrop that appeared in every reference.
Feature extraction. Pixels become numbers. A visual encoder produces a general appearance embedding, face-specific adapters add a sharper identity vector, and style encoders capture the texture and color of wardrobe and lighting. Different tools weight these channels differently, which is exactly why the same reference set can produce noticeably different results across models.
Aggregation. The per-image features are pooled into one profile. Many systems let you set per-image weights, so a clean front-facing portrait can dominate while a blurry candid contributes almost nothing. Keep one rule in mind: contradictory references do not cancel out, they average. Blend a bearded face with a clean-shaven one and you get someone vaguely related to both.
Conditioning injection. The profile meets generation here. Cross-attention layers consult it at every denoising step, while temporal layers hold it steady across frames. This is why a fused clip does not simply re-roll the face on every frame.
Layout control. Pose skeletons, depth maps, or edge maps constrain where the body and camera go, while the profile decides what that body looks like.
The takeaway is simple. Your reference set defines the ceiling of achievable consistency, and your prompts only steer within it. No clever wording rescues a contradictory set of references.
Preparing your reference set: the checklist that prevents most drift
Most disappointing results trace back to the references, not the model. Treat this checklist as a gate you must pass before generating anything.
- Count: six to twelve images per character. More only if they are genuinely varied. Thirty near-identical frames over-weight one angle.
- Angles: include a three-quarter view, a straight-on view, a profile, and at least one slightly low or high angle.
- Expressions: neutral, smiling, focused, worried. Expressions teach the identity adapter how the face deforms.
- Lighting variety: soft daylight, hard side light, warm interior, cool exterior. You are teaching the profile that this person survives a lighting change.
- Wardrobe consistency: if the character wears a green raincoat in the film, every reference should show that coat, or you should split references into per-costume sets.
- Resolution and sharpness: at least 1024 pixels on the short side. Motion blur and compression artifacts become part of the identity.
- Occlusion: avoid sunglasses, hands over the face, or scarves covering the chin in most frames. One or two are fine; a whole set is not.
- One character per image: group photos confuse isolation and often attach the wrong face to the profile.
- No heavy filters: strong color grading or beauty retouching fights the model's own color handling.
- Background diversity: a variety of locations in the references makes the profile generalize instead of baking in one room.
Locations need their own smaller sets: three to five wide shots of the same space from different positions, plus detail shots of distinctive props. Props that carry plot weight, a brass key, a specific lamp, deserve two or three isolated images so they can be referenced independently.
Sort your references by usefulness and name them clearly, for example elin_ref_front_neutral, elin_ref_threequarter_smile, elin_ref_side_hardlight. When a result is wrong, you will want to know which images fed it.
The shot bible: prompts that change only what should change
Once the profile exists, prompts should describe motion and framing, not appearance. Write a one-page shot bible before generating.
The bible has four parts. First, a locked anchor string repeated in every prompt: lens language, film stock feel, palette, grain, contrast, aspect ratio, and frame rate. Second, a character block that names the profile reference rather than describing the face. Third, a location block that references the location set. Fourth, a shot line that contains only what changes.
A usable template looks like this:
[profile: Elin] [location: lighthouse stairwell] [action] [camera move] [lighting and time of day] [locked style anchor]
Applied to four shots of one scene:
- Shot 01: Elin climbs the spiral stairs, slow dolly in, cold moonlight through a porthole, locked style anchor.
- Shot 02: Elin kneels beside a broken lamp, static medium shot, same moonlight, locked style anchor.
- Shot 03: close-up of her hands on the brass fittings, shallow depth of field, same moonlight, locked style anchor.
- Shot 04: Elin stands and looks up, handheld tilt, first hint of dawn, locked style anchor.
Notice what is absent: no age description, no hair color, no coat fabric. Those live in the profile. Repeating them in text competes with the references and creates a tug of war the model resolves unpredictably.
Keep a fixed seed whenever your tool allows it, and change one variable at a time when iterating. If a shot fails, do not rewrite the whole prompt. Tighten the action line, keep the anchor, and regenerate. Prompt churn is the fastest way to lose track of why a version worked.
Finally, number every asset. elin_sc03_take2 tells you more in the edit than final_final_new. Good naming is a continuity tool, not a filing habit.
A complete workflow: from stills to a finished scene
Step 1: Build the bible before you generate anything
Write the scene list, decide the locked style anchor, and collect references for every recurring element. This step takes an hour and saves days. Do not generate a single frame until the character sets and location sets are approved.
Step 2: Generate and approve keyframes as still images
Generate the first frame of each shot as a still, using either an image model or the video tool's first-frame mode. Judge these frames for identity, wardrobe, prop placement, and composition. Apparent motion quality is irrelevant at this stage, because you are only approving appearance. Reject anything that drifts before it costs you a full video render.
Lay the approved keyframes out as a contact sheet in shot order. Read the sheet as a comic strip. If the story does not read clearly in stills, it will not read clearly in motion.
Step 3: Lock the profile and stress-test it
Run three cheap tests: the character in a brand-new location, the character under different lighting, and the character at a different distance from camera. If the face holds across all three, the profile is stable. If not, swap in better references rather than pushing the identity weight higher, which tends to produce a stiff, waxy look.
Step 4: Animate shots in order, not in isolation
Generate shots in sequence and keep the previous approved clip visible while prompting the next one. Where your tool supports it, initialize the next shot from the last frame of the previous one. This chaining, sometimes called last-frame continuation, locks position, costume state, and lighting at the seam between shots, which is where audiences notice discontinuity most.
Keep each clip short, three to six seconds, and let the edit carry the pacing. Long generations drift more because the temporal layers have more frames in which to wander.
Step 5: Assemble, match, and repair
Bring everything into the editor and watch the sequence at full speed before judging frames. Drift that is obvious on a paused frame is often invisible in motion, so fix only what a viewer would actually notice. When something does break, repair the smallest possible element: regenerate the shot, not the sequence; inpaint the face, not the frame; adjust the grade on the offending clip rather than on all of them.
Choosing tools and models: decision criteria
Tool choice matters less than process, but the differences are real. Evaluate candidates against your specific film, not a demo reel.
| Criterion | What to test | Warning sign |
|---|---|---|
| Identity strength | Same character in five new locations | Face shifts subtly every shot |
| Reference count | How many images it accepts, and whether you can weight them | Only one reference slot |
| Layout control | Pose, depth, or edge inputs | No way to fix composition |
| Shot length | Quality at the duration you need | Sharp drop after two seconds |
| Style lock | Consistency of grade across clips | Colors shift between shots |
| Output resolution | Final delivery size | Upscaling required for every clip |
| Cost model | Predictable pricing for your volume | Unclear renewal terms |
| Iteration speed | Time from prompt to preview | Minutes per draft |
| Export and licensing | Commercial terms you can live with | Restrictions discovered late |
Run a one-scene pilot with each finalist: three shots, one character, one location change, one close-up. Then score identity, motion naturalness, and how much repair each shot needed. A tool that wins on faces but needs ten regenerations per shot loses to one that is slightly softer but reliable.
Also decide early between hosted and local generation. Hosted tools are faster to learn and handle heavy models for you. Local tools give you privacy, unlimited iteration, and full control over adapters, at the cost of hardware, setup time, and a steeper learning curve. Many teams use hosted tools for exploration and local tools for the final pass.
Mistakes that break continuity, and how to fix them
Too few references. One portrait cannot teach a model how a face moves. Fix: add angles and expressions until you have six to twelve images.
Conflicting references. Two different haircuts average into a third, wrong hairstyle. Fix: split references per costume or per act, never mix them in one profile.
Over-weighting identity. Pushing similarity settings too high freezes the face and kills expression. Fix: raise weight only until the identity holds, then stop.
Describing appearance in text. Text and references fight, and the model picks arbitrarily. Fix: move all appearance detail into the profile and keep prompts about action.
Generating out of order. A shot generated in isolation has nothing to anchor to. Fix: generate sequentially and chain last frames.
Ignoring aspect ratio mismatches. Cropping a reference to a new ratio can cut the head and confuse the encoder. Fix: prepare references in the target ratio before uploading.
Chasing every flaw. Not every drift matters on screen. Fix: prioritize the shots the audience will study, usually close-ups and the establishing shot.
No version discipline. Without naming rules you cannot tell which reference produced which result. Fix: name files and clips with scene, shot, and take numbers.
Skipping the stills pass. Animating before approving keyframes multiplies waste. Fix: approve all first frames as stills, then animate.
Advanced techniques for harder scenes
Pose and depth guidance. When the action matters more than the exact framing, drive the shot with a pose skeleton or a depth map. The layout controls the body, the profile controls the face, and the two stop fighting.
Multi-character scenes. Give each character its own profile set and generate them separately before combining. Trying to fuse two identities in one prompt usually produces a face that resembles neither.
Style locks across a whole film. Create a style reference from three or four frames you love, in the same palette, and reuse it in every prompt. A consistent grade hides small identity imperfections far better than a clean but wildly varying grade.
Last-frame chaining for moving scenes. For a character walking through a corridor, initialize each clip from the previous clip's final frame. Motion stays continuous and the costume cannot reset between shots.
Interpolation between keyframes. Generate two approved stills, an entry and an exit, then let the model interpolate between them. This is the closest thing to traditional keyframe animation and it gives you precise control over staging.
Selective separation of subject and light. Use the character profile for identity and a separate lighting reference for the mood, so a candlelit scene does not permanently tint the character's skin in the encoder's memory.
Palette transfer for scene transitions. Keep a single palette per act rather than per shot. Scenes feel like the same film when the color logic is consistent even if the location changes completely.
Quality control: catching drift before the edit
Build a review pass into the workflow rather than bolting it on at the end.
Watch each clip three times: once at full speed for rhythm, once paused on the first and last frames for continuity at the seams, and once at half speed for artifacts like warping hands or melting jewelry. Then place the clip against its neighbours in a rough sequence and check only three things: does the face read as the same person, does the outfit match, and does the light direction make sense from shot to shot.
Use a contact sheet of first frames for the whole scene. Mark each frame green, amber, or red. Fix reds first, ambers only if the shot is prominent. This simple triage prevents the endless polishing loop that kills small projects.
When repairing, prefer the least invasive fix. A grade adjustment can rescue a slightly warm shot. A single-frame inpaint can rescue a flickering eye. Only regenerate when identity itself has failed, and when you do, tighten references instead of adding adjectives.
Finally, archive each approved profile set and style anchor with the project. Reusable profiles make sequels, series, and branded content dramatically cheaper, because the hard part is already solved.
FAQ
How many reference images do I actually need?
Six to twelve varied images per character is the sweet spot. Fewer than five leaves the profile guessing about angles it has never seen. More than fifteen rarely helps unless the images add genuinely new information such as a second costume or a new expression range.
Can I use the same references for a stylized or animated look?
Yes, but keep the references in that same style. Mixing a photoreal portrait with an illustrated one produces an unstable average. If the film is illustrated, all references should be illustrated, ideally generated from one consistent character sheet.
Why does the face hold but the hands keep changing?
Most profiles are trained to prioritize faces. Hands, jewelry, and small props are usually better handled with a dedicated reference of the object and a prompt that names it explicitly, or by cropping tighter so the hands occupy more of the frame.
Should I animate from a single still or from multiple references?
Use both. Multiple references build the identity profile. A single approved still serves as the starting frame for the animation. The profile supplies who the character is, and the still supplies where the shot begins.
How do I keep lighting consistent between shots?
Write the light into the locked anchor, name its direction and quality, and reference a lighting image alongside your character set. Keep light direction stable across a scene even when intensity changes; audiences read direction more strongly than brightness.
What if the tool only accepts one reference image?
Composite several views into a single reference sheet, or composite the character onto a neutral background with a pose you want. It is a workaround, not a solution, so treat limited reference support as a hiring criterion for your next tool.
How long should each generated shot be?
Three to six seconds covers most coverage needs and keeps drift manageable. If a moment needs to breathe, generate two shorter clips and cut between them rather than one long clip that decays.
Do I still need a shot list if the model handles consistency?
More than ever. Fusion solves appearance, not storytelling. A shot list keeps you from generating beautiful, consistent, and completely unusable footage that never cuts together into a scene.
How do I handle a character who changes costume mid-film?
Build separate profile sets per costume and label them by act. Switching profiles at the right moment is cleaner than trying to force one profile to represent two looks, which usually averages them into a third look you never wanted.
What is the fastest way to improve results without new tools?
Improve the references. Sharper images, more angles, consistent wardrobe, and a cleaner background separation fix more problems than any settings change. After that, tighten prompts so they describe only action and camera, and generate shots in sequence with last-frame chaining. Those three habits alone move most projects from unusable to broadcast-ready, and they cost nothing but discipline.
