Text and Photos to Video: A Modern AI Workflow Guide

The front door to video production has moved

For most of the last century, making a video started with a camera and a location. You booked a shoot, gathered people and gear, and hoped the weather held. The edit came last, and every fix meant going back to the set. That sequence still exists, but it is no longer the only way in.

Today a surprising number of finished videos begin as a paragraph of text and a folder of still images. A script becomes a shot list, the shot list becomes a set of generated keyframes, and those keyframes become motion clips that get cut together with music and voiceover. No camera, no crew, no location scout — just a laptop, a clear plan, and a working knowledge of how these models behave when you push them.

This guide is a neutral, tool-agnostic playbook for that pipeline. It covers which generation paths to use for which job, how to choose between the many available video models, how to write prompts that survive a render, how to fix the failures that show up over and over, and how to organize the whole thing so you can repeat it next week. It deliberately avoids hype. The interesting question is not which model is "best" — it is which model is right for the specific shot you are trying to get.

The three generation paths, and when each one wins

Almost every AI video task falls into one of three categories. Choosing the right one early saves hours later.

Text-to-video

You describe a scene in words and the model generates motion from scratch. This is the most flexible path and the least predictable. It is ideal for establishing shots, abstract transitions, B-roll that does not need to match anything specific, and concept exploration during pre-production.

Its weakness is control. Characters drift, architecture morphs, and the model may invent details you never asked for. Text-to-video is a great sketchbook and a mediocre substitute for a locked storyboard.

Image-to-video

You supply a first frame — a generated still, a photograph, an illustration — and the model animates it. Because the composition, lighting, and identity are already fixed, the output is far more controllable. This is the workhorse path for narrative content, product shots, and anything where a specific face or object must remain recognizable.

Hybrid pipelines

The strongest results usually come from blending both. You might generate a still with a text-to-image model, refine it in a paint program, animate it with image-to-video, then use short text-to-video inserts for effects the animator cannot produce. Many teams also generate at a lower resolution, upscale, and then re-render selected frames at higher fidelity.

Path	Control	Speed	Best for
Text-to-video	Low to medium	Fast	Concepts, B-roll, transitions
Image-to-video	High	Medium	Narrative shots, products, characters
Hybrid	Highest	Slower	Anything client-facing or repeatable

Choosing a model by job, not by leaderboard

There is no single dominant video model, and chasing whichever one currently tops a public ranking is a losing game. Rankings measure averages. Your project needs one specific outcome: a believable product rotation, a talking head that stays on model, a stylized action beat. Match the model to the requirement.

Cinematic realism and physics

Several current systems — including the major commercial engines from Runway, Google's Veo line, OpenAI's Sora line, and Kling — are tuned for believable camera motion, depth, and light. They handle glass, water, and fabric reasonably well. Reach for these when the shot needs to look photographed rather than generated.

Stylized and illustration-friendly output

Pika, PixVerse, and several open-weight models handle anime, painterly, and graphic-novel aesthetics with fewer artifacts than realism-focused engines. If your brand look is flat and illustrative, testing a stylized model first will save you from fighting realism you never wanted.

Motion control and camera language

Some tools expose explicit camera instructions — dolly, crane, orbit, whip pan — and honor them more reliably than a prompt sentence will. Luma's systems and newer Kling releases are often used for this. When the shot is defined by camera movement rather than subject action, start here.

Open weights and local rendering

Wan, Hunyuan, LTX, and various Stable Video Diffusion derivatives can run locally or on rented GPUs. You trade convenience for control over privacy, cost structure at volume, and the ability to fine-tune. Teams handling sensitive footage often prefer this route for that reason alone.

Still-image models matter just as much

Because so much of the pipeline is image-to-video, your keyframe generator is arguably more important than your video model. Flux, SDXL derivatives, Midjourney, and Ideogram all produce strong frames. Pick based on how well the model handles hands, text, and consistent characters, since those are the three things that break most often downstream.

Writing prompts that survive the render

A prompt is not a wish. It is a compressed technical brief. Models respond best when you separate the elements of a shot instead of piling adjectives into one sentence.

Use a five-part structure

Subject: who or what, with two or three distinguishing details.
Action: one clear verb phrase. Not three.
Camera: framing, angle, and movement.
Light and atmosphere: time of day, source, weather, color temperature.
Style: film stock, lens, era, rendering approach.

A useful example: "A weathered fisherman in a yellow raincoat pulls a rope hand over hand, medium close-up, slow push in, overcast dawn light with wet haze, anamorphic lens, muted teal grade." Each clause maps to something the model can act on.

Keep motion modest

Ammateurs often ask for too much: a character walking, talking, turning, and gesturing across a crowded street in a six-second clip. The model tries to satisfy every instruction and produces mush. One primary action per clip, with secondary motion implied, almost always looks better.

Specify duration and aspect ratio deliberately

Six to ten seconds is the sweet spot for most engines. Vertical 9:16 for social, 16:9 for web and broadcast, 2.39:1 only if you enjoy cropping pain later. Decide before generating, not after.

Negative direction is underused

Many tools accept negative prompts, and even when they do not, phrasing matters. "No lens flare, no text overlays, no crowd" reduces the chance of unwanted elements. Consistent negative prompts across a project also make your output more coherent.

Test in cheap mode first

Generate three to five low-cost variations before committing to a high-fidelity render. You are looking for composition and motion, not final quality. This habit alone cuts a project's rendering time substantially.

Preparing stills that animate well

Image-to-video quality is largely determined before you press generate. A frame that looks beautiful as a still can be terrible as a first frame.

Compose for motion, not for a poster

Leave room in the direction the subject will move or the camera will travel. Keep the subject slightly off-center. Avoid compositions where a small limb is about to cross a complex background, because that is exactly where the model will smear.

Keep the first frame mid-action

A person standing neutrally with arms at their sides gives the model nothing to continue. A person mid-step, mid-turn, or mid-gesture gives it momentum. The generated clip will feel more alive because it is finishing a motion rather than inventing one.

Protect identity across shots

Character consistency is the hardest problem in this pipeline. Practical tactics that work:

Lock a seed when the tool allows it, so variation comes from your edits rather than randomness.
Build a small reference set — front, three-quarter, profile — and reuse it for every shot.
Keep wardrobe and hair identical between frames, including color temperature.
Avoid extreme expressions in keyframes; exaggerated faces are the first thing to warp.
Where available, use character-reference or subject-consistency features rather than relying on prompt text alone.

Match lighting between adjacent shots

If shot one is warm sunset and shot two is cool overcast, no amount of prompt discipline will make the cut feel seamless. Grade your keyframes before animating. It is far easier to fix a still than a clip.

A repeatable production workflow, step by step

The difference between a hobby experiment and a deliverable is process. Here is a sequence that works for everything from a fifteen-second ad to a three-minute explainer.

Step 1: Script to shot list

Write the script in plain language, then break it into shots of four to eight seconds. Each shot gets one line: what the audience must see, and how it moves. This shot list is your contract with yourself. Everything downstream references it.

Step 2: Storyboard with stills

Generate or sketch a still for every shot. Do not animate anything yet. Review the sequence as a slideshow with timing. Most structural problems — a jump in logic, a missing transition, a shot that lingers too long — are obvious at this stage and painful to fix later.

Step 3: Animate the keyframes

Work shot by shot, and resist the urge to batch everything at once. Animate the shot you understand best first; it teaches you what this model wants in this project. Once you have a template that works, reuse the phrasing and settings for similar shots.

Step 4: Assemble a rough cut

Drop the clips into an editor immediately, even at low resolution. Watch it with sound off, then with music on. You will discover pacing issues that are invisible when viewing clips individually.

Step 5: Fill gaps with text-to-video

Where the story needs a bridge — a skyline, a transition, a texture — generate it with text-to-video. These short inserts hide hard cuts and give the edit room to breathe.

Step 6: Sound, voice, and music

Voiceover generated with a text-to-speech tool should be recorded (well, generated) before the final edit, so you can cut picture to audio rather than the reverse. Music can come from a licensed library or a generative audio tool. Keep dialogue around -12 dB, music around -20 dB under speech, and duck the bed whenever someone talks.

Step 7: Grade and finish

Apply one look across the whole piece. A subtle film grain layer is a remarkably effective way to unify clips from different models, because it gives inconsistent micro-textures a common surface. Fix audio levels last, then export at your delivery resolution.

The failures you will hit, and how to fix them

Almost everyone runs into the same handful of problems. Knowing the fixes in advance saves entire afternoons.

Morphing faces and hands

This is the single most common artifact. Fixes: shorten the clip, reduce head movement in the prompt, downgrade to a wider shot so faces occupy fewer pixels, or animate from a still with a neutral expression. If a hand is in frame, consider reframing so it is not.

Flicker and texture crawl

Fine patterns — brickwork, foliage, fabric weave, metal grating — shimmer because the model regenerates them slightly differently each frame. Fixes: reduce the detail in your prompt, add slight motion blur, generate at a higher resolution and downscale, or apply temporal smoothing and denoising in post.

Unwanted camera drift

Models love to add a slow push. If you asked for a locked-off shot and got movement, say so explicitly: "static camera, no camera movement, tripod." Negative prompts help here too.

Text that turns to gibberish

Generative video still struggles with lettering. Add text in your editor or motion graphics tool instead of asking the model for it. If a sign must exist in the scene, keep it out of focus and use a clean plate.

Physics that betray you

Liquids that defy gravity, objects that pass through each other, footsteps that do not match the ground. Fixes: simplify the action, remove contact interactions from the prompt, or cut around the moment of contact. Often the best solution is to end the shot just before the physics would be tested.

Lip-sync mismatch

Generate or record the audio first, then drive the video from it using a dedicated lip-sync tool rather than hoping the base video model matches dialogue. Frame rate mismatches — 24 versus 25 versus 30 fps — cause subtle drift, so keep everything on one timeline standard.

Scaling from one video to a content system

Once a single video works, the temptation is to repeat the same manual steps forever. That does not scale. A few habits make the difference.

Name and version everything

A file named shot-014_v3_kling-seed8812.mp4 tells you more in six months than final_final.mp4 ever will. Store prompts alongside renders in a simple text file or spreadsheet. You will want to know what produced that one perfect shot — usually about three weeks after you have forgotten.

Build a reusable prompt library

Group prompts by shot type: establishing, product hero, dialogue close-up, transition. When a new project starts, you are adapting templates rather than starting from a blank page.

Keep a style bible

Palette, lens character, grade, grain, and typography rules in one short document. Consistency across a channel is what makes AI-assisted content feel intentional rather than assembled from spare parts.

Review in context

Never approve a clip on its own. Watch it in the timeline with the surrounding shots. Half the artifacts that look alarming in isolation are invisible in a cut, and half the clips that look perfect alone fall apart in sequence.

This part is not optional, and it is evolving quickly.

Use footage and images you have the right to use. Do not generate recognizable real people without consent, and be especially careful with public figures, minors, and anything that could be mistaken for a statement by a real person. Check the license terms of every model you use for commercial work — some permit it, some restrict it, and some require attribution. If your content includes synthetic voices or faces, label it where your platform or audience expects it. When in doubt, disclose. It costs nothing and protects everything.

Frequently asked questions

Do I need a powerful computer?

Not necessarily. Most commercial video models run in the cloud, so a mid-range laptop with a stable connection is enough. Local open-weight rendering does require a capable GPU with sufficient memory, but renting cloud compute by the hour is often cheaper for occasional projects.

How long should a generated clip be?

Five to ten seconds. Longer generations tend to lose coherence and drift in style. If a shot needs to run twenty seconds, generate two clips and join them on a motion-matched cut.

Is image-to-video always better than text-to-video?

For control, yes. For speed and exploration, no. Spend the extra two minutes preparing a keyframe when the shot matters; use text-to-video when you are still deciding what the shot should be.

How do I stop characters from changing between shots?

Fix the keyframes first. Same wardrobe, same hair, same lighting, similar framing, and reuse reference images. Then restrict camera movement so the model has fewer opportunities to reinvent the face.

Can I use this for client work?

Yes, with care. Confirm the commercial terms of each model, avoid generating recognizable people without permission, disclose synthetic media when appropriate, and always deliver a version you would be comfortable defending publicly.

Why does everything look slightly glossy and over-lit?

That is a bias in many training sets. Counter it by specifying light sources, asking for haze, grain, or a specific film stock, and grading down highlights and saturation in post.

What is the most common beginner mistake?

Trying to do too much in a single clip. One subject, one action, one camera move. Compose sequences, not super-shots.

How much post-production should I expect?

Budget roughly as much time editing as generating. Assembly, sound, and grading are where a pile of decent clips becomes a video someone wants to watch.

A short decision checklist

Before you start the next project, run through this list:

Is the goal exploration or a locked deliverable?
Which path fits each shot — text, image, or hybrid?
Does the chosen model handle your style and your subject reliably?
Are your keyframes consistent in lighting, wardrobe, and framing?
Is there exactly one primary action per clip?
Do you have audio designed before the final picture cut?
Are your files named and your prompts saved?
Have you reviewed the whole sequence, not just the individual clips?
Are the rights and disclosure questions answered?

Where this is heading

The trajectory is clear: more control, longer coherent sequences, better physics, and tighter integration between the still-image and motion stages. What will not change is the underlying craft. Story, pacing, framing, and sound design still decide whether anyone watches to the end. Generation models simply remove the logistical friction that used to sit between an idea and its first cut.

That is the real shift. Text and photos used to be pre-production artifacts — notes and references that pointed toward a shoot. Now they are the raw material. Treat the pipeline with the same discipline you would bring to a set: plan the shots, control the variables, review in context, and finish properly. Do that, and the tools stop being a novelty and start being a studio.