ControlFoley Adds Sound to Silent Video. The Mode Worth Your Time Is the One Nobody Demos.
A walkthrough of Xiaomi's video-to-audio model, the five modes it ships, the one that changes how you work, and the license line that decides whether any of it is usable on a paying job.
The 8-bit build of ControlFoley is 12.56GB. The model has about 4.99 billion parameters, so an honest 8-bit package should land somewhere near 5.3GB. The gap is not a rounding error, and the reason is worth knowing before you plan any local install around it.
I read the file's header directly. It holds 4,737 tensors, and exactly 1,093 of them are actually Q8_0. That is 23%. The vocoder stays fully 32-bit. The audio VAE stays fully 32-bit. So do 860 of the 1,238 tensors in the main network. audio.cpp, which produces these builds, documents the behavior plainly: q8_0 means "quantize eligible tensors to Q8_0; unsupported tensors remain in a backend-safe type." Most of this model's tensors are not eligible.
The quantization is real where it applies. The main diffusion network drops from 11.29GB to 8.15GB, a 28% cut. It just does not touch enough of the package to change the hardware conversation, which is the thing people actually want from a quantized build.
I lead with this because it kills the assumption most people arrive with. There is no lightweight path to running ControlFoley locally. There is still something here worth your afternoon, and it is not the thing on the box.
The problem it was built for
Video-to-audio models generate sound for silent footage. Most of them take the video, look at what is in it, and produce sound that matches. That works until you want something other than what the picture literally shows.
ControlFoley, from MiLM Plus at Xiaomi with collaborators at Wuhan University, is built around the case where your inputs disagree. The paper's own framing is cross-modal conflict handling. You hand it a clip of someone tapping a table and a prompt that says "thunder strike," and the design question is which input wins, and by how much.
That sounds like an edge case. For anyone cutting AI-generated footage it is the normal case. Generated video gives you motion without any recording of what that motion sounded like, and the sound you want is frequently not the sound the objects on screen would have made. You are art-directing audio against a picture that has opinions of its own.
The team built a benchmark specifically to measure this, which is the part I find persuasive. VGGSound-TVC grades text controllability at four conflict levels across five settings: L0 where the text agrees with the video, L1 split into subject swaps and action swaps, L2 where the text comes from a different category but keeps a similar rhythm, and L3 where the description is substituted at random. Building the measurement before claiming the result is the right order of operations. It is also their own benchmark measuring their own model, so read the numbers accordingly.
Five modes, and the ranking nobody gives you
The model ships five tasks. The documentation lists them as a flat table. They are not flat in value.
| Mode | Video | Reference audio | Prompt |
|---|---|---|---|
| V2A | required | none | none |
| TV2A | required | none | required |
| TC-V2A | required | none | required |
| AC-V2A | required | required | optional |
| T2A | none | none | required |
T2A is the throwaway. Text to audio with no video is a capability the framework gets for free, and there are better dedicated tools for it.
V2A is the demo mode. Point it at a clip, get sound. This is what every writeup screenshots and it is the least interesting thing the model does, because it is what every competitor also does.
TV2A and TC-V2A are the workhorses. Same inputs, different intent. TV2A is text guiding a sound that agrees with the picture. TC-V2A is text overriding a picture that disagrees. The command is identical; the difference is how far your prompt has strayed from what is on screen, and the model's handling of that distance is what the whole paper is about.
AC-V2A is the reason to care. You supply a reference audio file, and the model copies its timbre while keeping the timing locked to your video. The card describes the mechanism as extracting "global timbre representations while suppressing temporal cues, enabling precise acoustic style control without affecting synchronization." The reference contributes its character and contributes nothing about its rhythm.
Think about what that means at the desk. You have a sound you like, from a library or a previous project or a recording off your phone, and it is the wrong length with the wrong hits in the wrong places. Normally you are cutting and time-stretching it into your edit by hand. Here you hand it over as a description of what the sound should be like, and the model synchronizes to your footage instead. That is a different operation from anything a sample library gives you, and it appears in none of the coverage I read.
The authors report it beating CondFoleyGen, a specialist in-domain baseline, on the reference-audio task. That is their evaluation of their own model against a competitor, reported in their own paper, so treat it as a claim rather than a fact. It is the specific claim I would most want to test, and testing it is free.
How it hears the picture
Two visual encoders run in parallel. The card's wording: the model "combines CLIP and CAV-MAE-ST representations to capture both vision-language and audio-visual correlations, improving robustness under modality conflict." CLIP handles what the picture means. CAV-MAE-ST handles how sound and vision correlate over time, which is a different question and the one that governs whether a hit lands on the frame where the hammer strikes.
My reading of why that split matters, and it is my reading rather than a claim the team makes: if one encoder produced a single blended representation of "what this video is," then a prompt fighting the video would be fighting everything at once. With semantic meaning and temporal correlation on separate paths, text can override what the sound should be while the temporal path keeps deciding when it should happen. If that reading is right, pushing a TC-V2A prompt hard should give you a sound that changes category and stays on the beat. That is a prediction, not a result, and it is cheap to check.
Training added all-modality dropout alongside a unified REPA objective, so the model saw examples with inputs missing. The team credits that with robustness across modality combinations. I suspect it is also why one checkpoint covers five input shapes instead of needing five models, though they do not say so.
Put this into practice
Step one, and for many of you the only step: use a browser demo. It is free and needs nothing.
There are two official ones. Xiaomi runs a demo at yjx-research.github.io/ControlFoley_web_page/#try-gen, with tabs for four of the modes, a duration control, and a generate button wired to Xiaomi's own inference API. The Hugging Face Space at huggingface.co/spaces/YJX-Xiaomi/controlfoley runs the same model on ZeroGPU.
Worth correcting a thing repeated a few times this week, including in my own briefing feed: the Hugging Face Space, created August 14, was not the first no-install way to try this. The repository behind Xiaomi's demo page was created April 10, 2026, and by April 11 its built bundle already contained the upload inputs, the generate handler, and the calls to Xiaomi's /api/v1/v2a/submit endpoint. I pulled that April build artifact to check. The paper's arXiv v1 went up five days later, on April 16, which is also when the #try-gen deep link everyone shares was added.
One honest caveat on that: there is no Wayback snapshot of the page from any date, so I can prove the code existed in April and cannot independently prove GitHub Pages was serving it to the public that week. What I can say is that nothing about this model became newly usable in August. It became newly visible, which is a different thing.
Step two: test the mode that matters, not the one the demo opens on.
Upload eight seconds of your own footage, ideally something with two or three clear physical events in it. Run plain V2A first with no prompt to get your baseline. Then run AC-V2A with a reference sound whose character you want but whose rhythm is wrong. Compare where the hits land. That comparison, on your footage, in about ten minutes, answers the only question that matters, and you will have downloaded nothing.
Step three: if and only if the browser test convinced you, install locally.
git clone https://github.com/xiaomi-research/controlfoley
cd controlfoley
conda create -n controlfoley python=3.10.16
conda activate controlfoley
pip install -r requirements.txt
Before you run the weight download, know what you are pulling. The command in the README fetches the whole Hugging Face repository, which is 16.66GB, and about 80MB of that is demo media you do not need. The real payload is weights/controlfoley.pth at 11.29GB plus roughly 5.29GB of auxiliary encoders: a LAION-CLAP checkpoint at 2.35GB, a 44kHz audio VAE at 1.22GB, Synchformer at 950MB, and CAV-MAE-ST at 762MB.
That 16.66GB is not the finished total. Comparing the file list against the packaged GGUF build, which bundles every component the model needs, turns up four more that have no counterpart in the Hugging Face repository: OpenCLIP at nearly a billion parameters, plus BigVGAN, MERT, and a MusicGen style encoder. Those come down from elsewhere on first run. Budget closer to 20GB than 17.
The requirements file pins torch==2.5.1+cu121 with matching torchaudio, torchvision, and Triton. The official Python path is NVIDIA CUDA and nothing else: no CPU fallback, no Apple MPS, no ROCm.
Then the commands, one per mode:
# Sound that matches the picture, guided by words
python demo.py --video clip.mp4 --prompt "skateboarding" --duration 8.0 --output ./output
# Words overriding the picture
python demo.py --video clip.mp4 --prompt "thunder strike" --duration 6.0 --output ./output
# The one worth your time: borrow a sound's character, keep your timing
python demo.py --video clip.mp4 --audio reference.wav --duration 5.0 --output ./output
# No guidance at all, for a baseline
python demo.py --video clip.mp4 --duration 6.0 --output ./output
# Text only, no video
python demo.py --prompt "A bird sings melodically in a forest." --duration 10.0 --output ./output
Duration defaults to 8.0 seconds if you leave it off. There is a --negative_prompt flag that the docs list without explaining, worth an experiment if a particular unwanted sound keeps surfacing.
Step four, and the only route if you are not on an NVIDIA card: the C++ build. audio.cpp added ControlFoley in release 0.7.0, published August 27, and advertises Metal, Vulkan, CPU, and ROCm backends alongside CUDA. Two cautions. The v0.7.0 release notes on GitHub do not mention ControlFoley at all; the support is documented only in the project README's changelog and its supported-models table, and the GGUF files landed on Hugging Face five days earlier, on August 22. And ControlFoley is absent from audio.cpp's own per-model test-status table, while the project warns that backend and model coverage are model-dependent and that the non-CUDA backends are "intended for portability and testing." So a Mac path is advertised and plausible. It is not attested for this model, and I could not confirm it either way.
Where it breaks
The weights are CC BY-NC 4.0, and that is the end of the conversation for client work. The README's license line is easy to skim past because it names both: "This repository is licensed under the Apache License 2.0 and the model weights are licensed under the CC BY-NC 4.0." Repository, Apache. Weights, non-commercial. You cannot bill for output from this model no matter how good it sounds.
Two places you will see that stated wrongly. The Hugging Face Space's card metadata says apache-2.0, which describes the Gradio wrapper rather than the model inside it. More consequentially, the GGUF package README's redistribution table lists ControlFoley's original model license as Apache-2.0, on the same page that tells you redistribution is "governed by the corresponding original model license listed above." That table is wrong. Do not rely on it.
Where the NC term came from, I do not know. The acknowledgments list MovieGen-Audio-Bench among the datasets used, and that benchmark genuinely is CC BY-NC 4.0, and the team states that all resources are used for academic and non-commercial demonstration purposes only. It is tempting to conclude the license was inherited from the data. The team never says so, and evaluating on a non-commercial benchmark does not normally force non-commercial weights, so I am flagging that as a guess rather than a finding.
No hardware requirement is published. Not a VRAM figure, not an inference time, not a throughput number, not in the model card, the requirements file, or audio.cpp's performance tables, which cover a dozen other model families and skip this one. The only hard information is the file sizes and a CUDA 11.8 pin. The official Space runs on a ZeroGPU A10G, a 24GB card, which describes their deployment rather than a stated minimum. Any specific number you see quoted for this model, here or anywhere, is inferred. I am not going to invent one.
The ComfyUI integration is thin. The official node repository is six weeks old, has six stars, and has not been pushed to since July 30. It is not archived and has no open issues, so "abandoned" is too strong. Dormant is accurate. If your practice depends on node-graph tooling with an active maintainer behind it, this is not that.
Adoption is early. The Hugging Face model repository shows 143 downloads, which is a rolling thirty-day count rather than a lifetime total, and the code repository has 148 stars. For a paper accepted to ACM MM 2026 with a working browser demo and a C++ port, that is a small community, and it means you will not find much troubleshooting when something breaks.
It is built for short cues. Default duration is 8 seconds and the published examples run 5 to 10. This is a Foley tool for events, not a scoring tool for sequences.
A documentation bug, for your peace of mind. The arXiv badge on the model card displays the number 2506.21448, which belongs to ThinkSound, a different video-to-audio paper from a team at Alibaba, HKUST, and Zhejiang University. The link underneath the badge and the citation block both point correctly at 2604.15086, which is ControlFoley. It is a copy-paste error in a badge image, not a wrong citation. It briefly convinced me I had the wrong paper, so it will probably do the same to you.
What I would go find out
The claim I want someone to break is the timbre-without-rhythm one. On paper, suppressing temporal cues from a reference while keeping the video's timing is exactly the operation a sound editor performs by hand all day, and if it holds up on real footage it changes what a sample library is for. If it does not hold up, it will fail in a specific and legible way: either the reference drags its own rhythm along and the hits drift off the frame, or the timbre washes out into something generic. Both are obvious within a few seconds of playback.
That test costs nothing and needs no install. Take a clip you already cut, take the reference sound you fought with while cutting it, and put them both into the browser demo. Then tell me which way it broke, because the answer is more useful than another writeup of the demo mode.
The other thing worth watching is whether anyone ships a commercially licensed model with this architecture. Conflict handling is the good idea in this paper, and it is currently sitting behind a non-commercial line that nobody has explained. Good ideas do not usually stay behind one for long.
Medium metadata
Primary sources
- Model card: https://huggingface.co/YJX-Xiaomi/ControlFoley
- Paper: https://arxiv.org/abs/2604.15086
- Code: https://github.com/xiaomi-research/controlfoley
- Xiaomi browser demo: https://yjx-research.github.io/ControlFoley_web_page/
- Hugging Face Space: https://huggingface.co/spaces/YJX-Xiaomi/controlfoley
- audio.cpp: https://github.com/0xShug0/audio.cpp
- GGUF builds: https://huggingface.co/audio-cpp/audio.cpp-gguf/tree/main/ControlFoley-GGUF
File sizes, dates, licenses, and download counts verified against cache-busted Hugging Face and GitHub API responses on August 29, 2026. The tensor counts and quantization ratios come from reading the GGUF file headers directly. Download and star counts move; the Hugging Face figure is a rolling thirty-day count.