Ask any director who has finished a short film with generative video tools what caused the most rework, and the answer is almost never lighting or lens choice. It is the face. A character looks perfect in the opening scene, then walks back into frame ten minutes later with a slightly narrower jaw, a different eye colour, and an age that no longer matches the audio performance. The individual images are still beautiful. The story stops working, because the viewer quietly files that person as someone else.
Multi-image fusion is the craft answer to that problem. Instead of conditioning a model on one portrait and hoping the sampler behaves, you feed in a curated set of references and tune how strongly each one shapes the result. This guide covers the mechanics, the discipline of building reference packs, per-model tuning, a full shot-by-shot production workflow, compute planning, continuity review, troubleshooting, and the questions that come up most often when teams adopt the technique for the first time.
Why generative video breaks faces between shots
Every generated frame starts from noise. Unless identity information is carried forward deliberately, each shot is an independent roll of the dice. Small deviations are invisible in isolation and glaring once you cut shots together, because the human brain is extremely good at comparing faces across a cut. A two percent change in the distance between the eyes reads as normal variation in a photograph and as a casting change in a film.
Drift rarely appears randomly. It clusters in predictable places:
- Extreme angles. Profiles, low angles, and over-the-shoulder framings fail first, because the reference never showed the model that geometry.
- Lighting shifts. A face lit by soft window light does not survive a hard practical lamp without the model inventing new structure to justify the shadows.
- Expression peaks. Laughing, shouting, and crying reshape the face. Some models rebuild the whole head to accommodate an expression instead of moving the mouth and brow.
- Scale changes. A drift that reads as "slightly off" in a wide shot reads as a different actor in a close-up.
- Downstream assets. Wardrobe, hair, and props inherit the inconsistency, so a costume that matched in shot one suddenly looks like a variant from another production.
The cost is not only aesthetic. A failed shot usually means re-running a whole sequence, re-checking lip sync, re-grading, and re-editing sound. Consistency is cheap to enforce at the beginning of a project and expensive to repair at the end. It is also the main difference between a demo reel that impresses for thirty seconds and a sequence an audience will sit through without losing the thread.
What multi-image fusion actually does
Fusion combines several reference inputs into one conditioning signal, weighted so that identity traits dominate while incidental details fade. The model still samples freely for pose, motion, and lighting, but its freedom is constrained by the merged reference. A useful mental model: you are not describing a photograph of a face, you are casting a role and handing the model the audition materials.
Identity conditioning versus style conditioning
A single reference image is ambiguous. The model cannot tell whether you want the person, the colour grade, the lens, or the background. It usually assumes all of them, which is why one-portrait workflows make every shot look like it was photographed in the same room at the same hour.
Stacking references resolves that ambiguity. Several angles of the same face tell the model what is invariant about the character. The differences between those frames tell it what is incidental. In practice you get a stronger identity anchor and much less accidental copying of a reference's mood, palette, or composition.
Why variation between references matters more than volume
Ten near-identical frontal portraits teach the model nothing new. Each additional image should add information the pack does not already contain: a new angle, a new lighting condition, a new expression, or a new wardrobe state. A pack of eight carefully differentiated images will outperform forty loose photos from the same photoshoot, and it will render faster because the conditioning signal is cleaner and less contradictory.
What fusion cannot fix
Fusion stabilises appearance. It does not fix blocking, timing, performance, or story logic. If a character turns left in one shot and arrives from the right in the next, no reference pack will save the cut. Treat identity lock as the foundation of a scene, not the finished result, and expect to direct motion, eyelines, and pacing separately.
Building a reference pack that survives every angle
A reference pack is a small, deliberate dataset. Coverage beats quantity, and consistency inside the pack beats both.
The coverage matrix
For a speaking role in a short film, aim for eight to twelve images that share one identity but differ in camera and light:
- One clean frontal portrait with a neutral expression and even lighting as the anchor image.
- Two three-quarter views, left and right, to define cheekbone and jaw structure.
- One true profile to lock the nose bridge, chin projection, and ear placement.
- One slight low angle and one slight high angle, so the model has seen the face from below and above before you ask for a dramatic tilt.
- One expressive frame, laughing or mid-speech, so expression is associated with identity rather than replacing it.
- One half-body or full-body frame for proportions, posture, and silhouette.
- Optional wardrobe plates, shot under lighting similar to your scene, if a costume must match exactly.
If your coverage plan includes shots the pack does not anticipate, add a reference now rather than discovering the gap after twenty renders.
Lighting coverage is not optional
A pack shot entirely under soft daylight will struggle in a night exterior or a fluorescent interior. Include at least one reference with directional light and one with mixed colour temperature. You are teaching the model how this face behaves when light changes, which reduces the temptation to redesign the skull in order to explain a shadow.
What weakens a pack
Heavy beauty retouching erases the skin texture that carries identity. Sunglasses, hands on the face, and strong shadows hide structure. Busy backgrounds leak into outputs. Duplicates create the illusion of thoroughness while adding nothing. Cull ruthlessly and keep notes on which images survived, because the pack will be versioned and reused.
Writing a character bible the model can actually use
A character bible is one page of plain text that describes who the character is and how they should be rendered. Its function is to keep every prompt, in every session, aligned with the same intent. Without it, subtle wording changes between sessions quietly drift the face.
Layer the attributes
Split your description into three layers. Locked attributes never change: age range, build, hair colour and length, skin tone, eye colour, distinctive marks, and permanent wardrobe pieces. Semi-variable attributes change between scenes: outerwear, hair styling, injuries, dirt, sweat. Free attributes belong to the scene only: lens, palette, grain, lighting direction, weather.
Write locked attributes once and paste them verbatim into every prompt. Changing word order or synonyms is enough to shift a result, so consistency in language matters as much as consistency in images.
Keep prompt scaffolds small and repeatable
A practical scaffold looks like this: locked identity block, then scene block, then shot block, then technical block. Keep each block short. Long prompts with conflicting adjectives are a common cause of drift, because the model has to decide which instruction wins. When a shot fails, change one block at a time so you can attribute the result.
Record what worked
Store the pack version, the prompt scaffold, the model and checkpoint version, the reference weights, and the seed family for every approved shot. A living lookbook of approved stills, one per camera setup, will save more time than any single setting tweak.
Tuning fusion strength by model family
Most tools expose some form of reference strength or influence control. Start near the middle of the range, raise it until features and wardrobe lock reliably, then reduce it slightly if motion stiffens or frames start looking copied from the reference.
Diffusion-based image pipelines
Diffusion models respond well to stacked references at moderate weights. Push the weight too high and frames become rigid recreations of reference compositions, including pose and background. Two characters sharing one conditioning slot may inherit each other's features, so give each character their own reference set and avoid blending them in a single pass. Keep style references in a separate, lower-weight slot so wardrobe and palette do not fight identity.
Transformer and temporal-attention video models
Video models with temporal attention carry information between frames, so identity usually holds within a shot and breaks between shots. Fusion here is mostly about shot-to-shot continuity: reuse the same pack, the same prompt scaffold, and the same seed family, and vary only the elements that must change. Higher reference weights help, but watch for ghosting on fast motion, where the model averages two subtly different faces into an ambiguous third.
Hybrid pipelines
A common and reliable pattern is to generate a still frame per setup with an image model, verify it against the approved lookbook, then animate that still with a video model using the same pack as a secondary reference. This separates identity decisions from motion decisions, which makes failures easier to diagnose. When a shot breaks, you know whether the still was wrong or the animation wandered.
A shot-by-shot production workflow
A repeatable sequence matters more than any individual setting. The following order has proven practical for short films, trailers, and branded narrative pieces.
- Write the character bible. One page per speaking role, with locked attributes listed explicitly.
- Generate and cull the reference pack. Produce far more candidate images than you need, then keep only the differentiated ones.
- Version the pack. Save it with a date and a short note describing what changed. Never edit a pack silently mid-production.
- Block the scene on paper. For each shot, note camera angle, lens, lighting direction, wardrobe state, and the emotional beat.
- Produce one look-dev still per setup. Approve it before animating. This is the cheapest place to catch a missing angle.
- Animate in small batches. Two or three related shots at a time, with identical pack and seed settings, so drift is visible before it spreads.
- Assemble an early cut immediately. Drift that is invisible in a contact sheet becomes obvious on a timeline with matching action and dialogue.
- Fix by exception. When a shot fails, change one variable, re-render at draft quality, and compare against the approved still from the previous shot.
- Lock, then finish. Upscale, restore detail, and grade only after the edit is locked, because grades hide small identity differences until they do not.
A worked example
Imagine a three-shot dialogue scene in a kitchen: a wide establishing shot, an over-the-shoulder shot of the protagonist, and a close-up reaction. The wide shot is forgiving and generally passes first. The over-the-shoulder shot is where drift usually appears, because the model sees a partial face and invents the rest. The close-up amplifies whatever the first two shots established.
With fusion in place, you would generate a look-dev still for each angle using the same pack. If the over-the-shoulder still shows a different ear shape or hairline, add a rear three-quarter reference to the pack rather than wrestling with the prompt. Then animate the three shots together as one batch, with the batch rendered at draft resolution, so the identity stays anchored across the sequence.
Planning compute around consistency, not resolution
It is tempting to render everything at maximum resolution. That is usually the wrong place to spend processing time. Consistency problems are cheap to detect at low resolution and expensive to repair at high resolution, so iterate at draft quality and upscale only approved shots.
A workable pattern:
- Draft tier: the lowest resolution where facial structure is still readable. Use it for all continuity checks.
- Preview tier: a mid-resolution pass for cut assembly and client review.
- Final tier: upscaling and detail restoration for approved shots only.
- Batching: group shots that share lighting and wardrobe so the same references stay loaded and settings stay identical.
- Caching: keep approved stills and intermediate frames so a failed animation pass does not force a full re-render from scratch.
- Naming conventions: encode pack version, model version, reference weight, and seed family in filenames so any frame can be traced back to its settings.
When a schedule tightens, cut the number of shots, not the number of references. A shorter film with a stable cast reads as professional. A longer film with a shifting face reads as a test render, no matter how good the lighting is.
Continuity review: a protocol, not a vibe check
Review the cast, not the shots. Watch a silent cut and note the exact frame where a character stops looking like themselves. Most teams find three recurring culprits: angle changes, lighting changes, and emotional peaks. Fixing those three categories resolves the majority of drift.
A short checklist for each pass:
- Does the silhouette match the previous shot?
- Do eye colour, hairline, and brow shape hold across cuts?
- Does wardrobe hardware, such as buttons, straps, or jewellery, stay identical?
- Do skin tone and texture survive the grade in both shots?
- Does the character still look like themselves mid-expression?
Use an escalation ladder instead of guesswork. First, re-render the same shot with the same settings to check for randomness. Second, add the missing reference that covers the failing variable. Third, adjust reference weight. Fourth, change camera framing to something the pack already covers. Fifth, escalate to face restoration or manual paint work in post. Stopping at step two saves enormous time; most teams jump straight to dramatic measures.
Troubleshooting common failure modes
| Symptom | Likely cause | Practical fix |
|---|---|---|
| Character appears to age between shots | Weak or inconsistent pack | Add two new angles, raise reference weight, reuse the seed family |
| Face holds but wardrobe drifts | Costume described only in text | Add wardrobe plates to the pack or lock the costume as a separate reference |
| Every frame looks like the reference photo | Style and identity mixed in one set | Split style references from identity references and lower style weight |
| Motion looks stiff or frozen | Fusion strength too high | Reduce weight, lower temporal consistency, re-animate the shot |
| Two characters blend features | Shared conditioning slot | Generate separately, composite in post, or mask each character |
| Details melt during fast action | Temporal attention overloaded | Slow the action, shorten the shot, add motion blur in post |
| Identity survives but emotion reads flat | Expression missing from the pack | Add one expressive reference and describe the beat concretely |
Log every fix with the pack version it belongs to. A troubleshooting table is only useful if it becomes a project record rather than a document nobody updates.
Extending the system to longer projects
Series work, episodic content, and recurring brand characters all benefit from the same discipline, with two additions. First, freeze packs at milestones and treat any change as a new version with a clear changelog. Second, build a per-character shot library of approved stills, tagged by angle and lighting, so new setups can be matched against existing coverage before rendering begins.
For episodic production, consider a character sheet that specifies not just the face but the wardrobe continuity rules across episodes, including when a jacket is worn open, when hair is tied back, and which props are permanent. These details cause more continuity complaints than facial structure once a series passes its first few episodes.
FAQ
How many reference images are enough for a main character?
Eight differentiated images usually beat twenty random ones. Cover front, both three-quarter views, profile, two vertical angles, one expression, and one body frame. Add wardrobe plates only if the costume must match exactly across shots.
Can a character stay consistent across multiple projects?
Yes, if the pack is versioned and frozen. Export the settings that produced approved frames, treat any edit as a new version, and never tweak a live pack in the middle of a sequence. Silent changes are how consistency quietly disappears.
Should I use face restoration tools on top of fusion?
Sometimes, and ideally as a repair step rather than the main method. Fusion solves identity at generation time; restoration is useful for extreme angles or imperfect source material. Overuse produces the slightly plastic look that audiences associate with over-processed footage.
Why does the same pack produce different results on different days?
Model versions change, sampler settings drift, and prompt order affects outcomes. Pin model versions, log full parameter sets, and store approved stills as reference points so you can compare new renders against a known standard.
Is this workflow realistic for a solo creator?
Yes, with scope discipline. A one-page character bible, a ten-image pack, draft-resolution iteration, and batching two or three shots at a time is a realistic weekend workflow for a short scene.
Does fusion replace casting and performance?
No. It stabilises appearance so the audience keeps believing in one person. Performance still comes from how you write action, direct eyelines, and cut the scene. Identity lock is the floor, not the ceiling.
What is the single biggest mistake teams make?
Changing multiple variables at once. If you alter the prompt, the reference weight, and the pack simultaneously, you cannot tell which change fixed the shot, and you will not be able to reproduce it later.
How do I handle crowd scenes or background characters?
Lower the bar deliberately. Background faces do not need individual packs, but they do need to avoid looking like distorted versions of your leads. Generate crowds separately, keep them out of close-ups, and use depth of field to reduce how much detail the audience reads.
A final pre-render checklist
Before you commit processing time to a scene, confirm that the character bible is written, the pack is versioned and covers every angle in your coverage plan, look-dev stills are approved for each setup, and the draft pass is complete. Then render in small batches, compare each new frame against the approved still from the previous shot, and stop the moment a frame feels like a different person. Consistency is not a plugin you install once. It is a habit of checking early, in small batches, against a written standard that everyone on the project can read.
