AI Audio Cleanup for Video: A Practical Post-Production Workflow

Every video editor learns the same lesson eventually: audiences forgive soft focus, imperfect framing, even a slightly shaky handheld shot, but they leave within seconds when dialogue sounds hollow, hissy, or distant. Audio quality is the fastest read on production value, and historically it was also the hardest thing to repair after the shoot. That balance has shifted. Modern AI cleanup tools can separate a voice from passing traffic, rebuild a room that never existed, and pull intelligible dialogue out of a phone recording made in a stairwell.

This guide is a practical workflow rather than a product tour. It covers how the models actually work, which tool fits which defect, the order of operations that keeps processing transparent, and the quality-control habits that stop cleanup from quietly becoming damage.

How AI Cleanup Models Differ From Classic Filters

Classic audio repair relied on a small set of predictable moves. Spectral subtraction estimates a noise profile from a silent passage and subtracts that energy from the whole file. It works up to a point, but subtraction is blunt: it also removes energy in the frequency bands where the voice lives, which produces the watery, hollow texture engineers call musical noise. Gates and expanders add a second failure mode, cutting abruptly on every breath so the background pumps in and out like a tide.

Deep-learning models attack the problem from a different direction. Instead of attenuating energy, they predict what the clean signal should have been and reconstruct it.

Reconstruction instead of subtraction

A typical modern denoiser is trained on paired examples: the same recording clean, and the same recording degraded by fan noise, rain, keyboard clatter, traffic, or room reflections. The network learns a mapping from noisy spectrogram to clean spectrogram. Architectures vary. Modified U-Net designs with skip connections remain workhorses for speech, while attention-based and transformer variants handle longer context and preserve consonants better at the edges of words. The practical consequence is that the model can rebuild a consonant it never cleanly heard, using learned priors about how human speech behaves.

This is why two tools can look nearly identical in their interface yet sound completely different on the same file. A model trained mostly on studio voice will struggle with a lav mic rubbing against a jacket. A model trained on field recordings handles that rustle but may over-smooth a studio voice into something synthetic.

What the model is actually listening for

A useful mental model groups defects into four families, because each maps to a different processing stage:

• Tonality: hum, electrical buzz, whine, ground loops. Predictable and narrowband, so it responds well to notch filtering and harmonic tracking.
• Broadband noise: fans, air conditioning, hiss, wind, distant traffic. Constant in character, so it responds to spectral or generative denoising.
• Transient events: clicks, chair scrapes, keyboard taps, lip smacks, plosive pops. Short and impulsive, so they need repair tools rather than noise reduction.
• Space: reverb, slapback echo, boxy small rooms, gymnasium reflections. These need dereverberation, and more denoising will never fix them.

Most cleanup disasters come from using one stage to solve another family of problem. Pushing a denoiser to maximum to fight a bad room is the single most common mistake, and it produces the metallic, chirpy artifacts that make a track feel worse than the original noise.

Choosing the Right Tool for Each Audio Problem

Start by naming the defect precisely. Vague intentions like make it sound professional lead to stacking plugins until the voice disappears.

Problem	Best first move	Risky shortcut
Constant hiss or fan noise	Broadband denoise at moderate strength, two light passes	Heavy noise gate
Electrical hum or buzz	Harmonic notch at the mains frequency and its multiples	Broadband denoise alone
Boomy, echoey room	Dedicated dereverb or speech enhancement pass	More denoising
Music or crowd bleeding over dialogue	Source separation into stems	Aggressive EQ carving
Clipped, distorted peaks	Declip or restoration model	Hard limiter
Codec artifacts from compressed sources	Bandwidth extension plus light denoise	Heavy high-frequency EQ
Plosives on p and b sounds	De-plosive tool or manual clip gain	High-pass at 120 Hz
Harsh sibilance	De-esser in the 4 to 8 kHz range	Denoise aimed at high frequencies

Decision criteria matter more than the table. Ask four questions before you process anything:

1. How much tonal detail can this content afford to lose? A podcast survives mild smoothing. A singing performance does not.
2. Do I have an alternate take or a second microphone? If yes, repair less and cut more.
3. Will the audience see lips move on screen? Tight sync tolerance means you should avoid tools that shift timing or phase.
4. Is the deliverable voice-forward or music-forward? Speech models should never touch a music bed.

The Order of Operations That Prevents Artifacts

The sequence below works for interviews, voiceovers, podcasts, documentary scenes, and corporate video. It assumes one dialogue track that needs rescue.

1. Edit first. Cut out unusable material and alternate takes before processing. Never clean audio you are going to delete.
2. Gain stage. Bring the raw track to a comfortable working level, roughly minus 18 dBFS average, so plugins behave consistently.
3. Repair transients. Remove clicks, pops, plosives, and mouth noise. These are short events and easy to miss later once broadband processing has softened them into the surrounding sound.
4. Remove tonal interference. De-hum at 50 or 60 Hz plus harmonics. Do this before denoising, because a denoiser will smear hum into the surrounding spectrum and make it harder to track.
5. Denoise in two gentle passes. Two passes at 40 to 60 percent strength almost always beat one pass at full strength, because each pass has an easier estimation problem.
6. Dereverb. Only now, on an already-cleaner signal, do reflections become clearly separable from the voice.
7. Tonal shaping. High-pass around 80 to 100 Hz, control mud around 200 to 400 Hz, add gentle presence around 2 to 4 kHz.
8. Dynamics. Compression at 2:1 to 4:1 with 3 to 6 dB of gain reduction, then a de-esser after compression rather than before.
9. Loudness. Normalize to your platform target last, because every earlier stage changes the level.

The reason order matters is that each stage changes the estimation problem for the next one. Denoising before de-hum leaves the model guessing about a tonal component it cannot easily distinguish from speech harmonics. Dereverbing before denoising makes the model chase reflections that are still buried under broadband noise. Following the chain above keeps every decision legible, and legible chains are the ones you can troubleshoot.

Keep everything non-destructive. Save the raw file untouched, render intermediate stages to separate tracks, and label them. When a client asks for a warmer version, you should be adjusting one stage, not redoing the whole chain by ear.

Source Separation: Turning One Mixed Track Into Editable Stems

Transformer-based source separation has quietly become one of the most useful tools in video post. Instead of processing a mix as a single stream, the model estimates the components that produced it: dialogue, music, ambience, and sometimes individual instruments or voices.

Why this matters for video work:

• You can recover dialogue buried under a licensed music bed without losing the music entirely.
• You can isolate a single lav track from a scene where two microphones bled into each other.
• You can remove an unwanted background loop from a location recording and replace it with clean ambience.
• You can duck music automatically during speech with genuine separation rather than a sidechain compressor guessing.

Separation is not free. It introduces its own artifacts, usually audible as a faint pre-echo, a slightly hollow timbre, or a residual ghost of the removed element. Three rules keep it usable:

First, separate once at the highest quality setting available, then edit the stems. Running separation on an already-separated stem compounds artifacts fast.

Second, check phase. Recombining stems can cause cancellation if anything shifted in time. Sum the stems back together, invert one channel, and listen. If the combination is not silent, something has moved.

Third, keep the original as a fail-safe. Some passages will sound better with the original mix than with any separated version, and you want to be able to swap a section back without rerunning the entire job.

Voice Restoration Checklist for Interviews, Voiceovers, and Podcasts

Once the general chain is in place, run through this checklist. It is ordered from structural fixes to cosmetic ones.

• Hum removal. Verify the mains frequency for the region where the recording was made, then notch that frequency and its harmonics.
• Broadband denoise. Two moderate passes, checking the tails between words for pumping.
• Plosive repair. Handle the worst pops with clip gain automation or a dedicated de-plosive tool rather than a blanket high-pass.
• Mouth noise and clicks. Auto-detection works well on isolated clicks, but manual removal catches the ones the detector misses.
• Sibilance. De-ess after compression so the compressor does not exaggerate the very frequencies you are trying to tame.
• Breath management. Do not gate breaths out entirely. Attenuate them 6 to 10 dB so the performance still sounds human.
• Tonal shaping. High-pass, a small cut where the voice sounds muddy, and a modest presence lift.
• Dynamics. Gentle compression for consistency, plus a limiter as a safety net rather than a loudness tool.
• Loudness. Normalize to target and confirm true peak headroom.

Room Tone and Reverb: Rebuilding Space Instead of Erasing It

Dereverberation is the stage where taste matters most. Removing every reflection produces a voice that sounds vacuum-sealed and uncanny, especially when cut against a wide shot of a real room. The goal is not a reflection-free track, it is a track whose space matches the picture.

Two techniques keep the result believable. First, capture or synthesize a matched room tone bed. Sample two or three seconds of clean ambience from the recording, loop it under the dialogue, and use it to smooth edit seams and hide processing transitions. If the original has no clean ambience, generative ambience tools can produce a neutral bed that sits under dialogue without drawing attention.

Second, add back a small amount of early reflection. A short reverb between 0.2 and 0.5 seconds, mixed 15 to 20 dB below the dialogue, glues automated dialogue replacement lines to production audio and softens the dryness that aggressive dereverb leaves behind. When you are matching two microphones in the same scene, treat the reverb bus as the shared space and let both sources feed it.

Also watch for the boxy midrange that small rooms create. A narrow cut somewhere between 300 and 500 Hz often does more for intelligibility than another denoise pass.

Loudness, Dynamics, and Delivery Targets for Video

Loudness is where technically clean audio still fails, because platform expectations are not the same as studio expectations. Common integrated loudness targets are around minus 14 LUFS for streaming video and music platforms, around minus 16 LUFS for spoken-word podcast delivery, and minus 23 or minus 24 LKFS for broadcast standards. True peak ceilings generally sit near minus 1 dBTP, and closer to minus 2 dBTP when the file will be encoded to a lossy format.

Use a loudness meter, not a peak meter. Peak meters tell you nothing about perceived level, and they encourage crushing dynamics to make numbers look bigger.

Three practical habits help. Keep dialogue 6 to 10 dB above the music bed so speech stays intelligible on phone speakers. Check mono compatibility, because a lot of cleanup processing is not perfectly symmetrical and phase issues hide in stereo. And leave the limiter doing almost nothing. If your limiter is working more than a decibel or two on dialogue, the problem is compression upstream.

Quality Control: A Listening Protocol That Catches Damage

The hardest part of AI cleanup is hearing your own damage. Build a repeatable protocol instead of trusting a single casual listen.

• Listen on three systems: closed headphones, studio monitors or decent speakers, and a phone speaker. Phone speakers expose muddiness and lost consonants faster than anything else.
• Solo and push the level. Turn the processed dialogue up louder than comfortable for 20 seconds. Artifacts like metallic ringing, chirping, and faint water sounds become obvious.
• Bypass and compare short passages. A five-second A/B on a difficult word reveals more than a full playback.
• Listen to the tails. The space between words is where denoisers pump and where separated stems leave ghosts.
• Check consonants. T and S sounds disappear first when a model over-smooths, and their absence is what makes cleaned audio sound mushy.
• Verify sync. Any time-domain processing can shift phase, so re-check lip sync after separation and dereverb.
• Inspect the seams. Edit points and crossfades are where room tone beds and reverb buses most often mismatch.
• Document the chain. Save presets, write down parameter values, and keep versioned files so a later revision request does not send you back to square one.

Common Mistakes That Undo Good Cleanup

The most expensive mistakes are consistent across skill levels.

Over-denoising is the biggest one. If a track sounds even slightly underwater, back off by 20 percent. Listeners tolerate a little steady hiss far more easily than they tolerate a hollow, artificial voice.

Using AI as a substitute for microphone technique is the second. A lav placed correctly under the chin with a wind shield beats any restoration model applied to a mic buried in a shirt collar. Fix the next shoot, not just this file.

Gating breaths out is the third. Silence between phrases feels robotic, and the pumping that gates create is audible on every phone speaker.

Other frequent errors include fixing loudness before cleanup, running speech models across a music bed, ignoring phase when recombining separated stems, restoring audio that was already delivered at low bitrate, and never listening to the finished piece end to end at normal speed. That last one catches problems no meter will ever show.

Frequently Asked Questions

Can AI cleanup rescue audio recorded on a phone?

Often yes, up to a point. Models handle steady noise, limited bandwidth, and moderate room reflections well. They struggle with severe clipping, wind hitting the microphone directly, and recordings where the voice is quieter than the surrounding environment. If the original is unintelligible to a human listener, no restoration tool will reliably invent the missing words.

How much noise reduction is too much?

If you can hear the noise reduction as a texture, it is too much. Practical ceilings look like two passes at 40 to 60 percent strength rather than one at full power. Judge by the space between words and by how natural the ends of phrases sound.

Should I denoise before or after editing?

After. Cut first, clean second. Processing audio you are going to remove wastes time and can pull artifacts into the edit decision process, since you start choosing takes based on how they sound cleaned rather than how they actually sound.

Does source separation always improve dialogue?

No. Separation is powerful when an element genuinely overlaps the voice, such as music or traffic. On a clean recording it can add subtle hollowness for no benefit. Test one scene before committing an entire project to a separated workflow.

Can I remove reverb completely?

You can reduce it substantially, but removing all of it sounds unnatural, especially against a wide shot. Aim for a level of space that matches the picture, and consider adding a small, controlled early reflection back for realism.

What loudness target should I use for a client video?

Start from the destination platform. Around minus 14 LUFS integrated for general streaming video, minus 16 LUFS for spoken-word podcast delivery, and minus 23 or minus 24 LKFS for broadcast. Keep true peak at or below minus 1 dBTP, and lower when the deliverable will be compressed lossily. When a client has no spec, deliver a version that meets the platform norm and document the settings you used.

The broader lesson is that AI cleanup is an assistant, not a decision maker. Models are excellent at estimating what a clean signal probably looked like, and they are indifferent to whether the result serves the story. Your job is to decide how much of the original to preserve. Handle the structural problems first, keep each stage gentle and reversible, verify with a listening protocol rather than a meter alone, and treat the final few percent of repair as a creative choice. Do that and a difficult recording stops being a liability, becoming just another track you know exactly how to finish.