AuK Makes the Sentence the API, and Its Own Docs Cannot Keep the Sentences Straight
A 1.5B MIT-licensed speech model does sixteen jobs through one text box. That design puts unusual weight on getting the words right, and the words in the documentation disagree with each other.
Tencent Hunyuan's AuK has no task dropdown. There is no mode flag, no --task separate, no radio button for "denoise" versus "de-accent." You hand it an audio file and a sentence, and the sentence decides everything: whether it clones a voice, strips a room's reverb, pulls one speaker out of a crowd, rewrites a lyric, or turns a whisper into normal speech. Sixteen documented jobs, one text field.
That is a real design choice with a real consequence, and the consequence is not the one the announcement talks about. When the instruction is the interface, the phrasing of that instruction becomes a tuning parameter as load-bearing as a sampler or a CFG scale. Which means the documentation is not a convenience. It is the parameter reference. And AuK's documentation says different things in different files about the same task, sometimes on the same input file.
I went looking for the gap between what the model promises and what the docs actually specify, and it is wider than one bad example.
What the thing is, before the criticism
Credit first, because the criticism only matters if the model is worth using.
AuK is a 1.5 billion parameter speech foundation model released under a plain MIT license. Not a research license, not a community license with a revenue ceiling, not a territory clause. MIT. The Hugging Face repositories were created August 18 and August 21, and the project's own News line dates the open-source announcement to September 9, 2026. The technical report is arXiv:2609.08936, authored by a list of thirty-odd researchers across Tencent Hunyuan and several universities.
The sixteen tasks span five categories: zero-shot and instruction TTS; content editing that replaces, inserts or removes words; acoustic editing for pitch, speed and volume; paralinguistic editing for emotion, timbre, accent, nonverbal sounds and whisper conversion; and enhancement plus separation, covering denoising, dereverberation, speaker separation, music separation and target-speaker extraction. Two variants ship: base, which runs 32 Euler steps with guidance, and Flash, a distilled version pinned at four steps with guidance disabled.
You can try all of it free right now. The official Space is running on zero-a10g hardware with demo inputs for Sichuan and Tibetan accents, content edits in both languages, whisper conversion in both directions, and vocal extraction. No install, no key.
And it runs small. A third-party ComfyUI repack publishes the diffusion transformer at 880 megabytes in W4A8 alongside a 3.179 GB quantized encoder and a 637 MB VAE that stays in FP32 on purpose. One reported peak with that quantized pair was around 8 GB, which the packager labels an observation rather than a floor. On Apple Silicon, an MLX port posted September 12 claims the four-step pass on ten seconds of audio drops from 3.820 seconds under PyTorch MPS to 0.992 seconds, with 8-bit backbone memory falling from 5.70 GB to 0.56 GB. Those are the author's numbers, on unstated hardware, in a repository with zero stars. Treat them as a claim.
The mechanism, and why phrasing carries so much weight
Here is what actually happens when you type a sentence, because it explains why the sentence matters so much.
Your instruction and your audio go in as a single ChatML-style message: a text block and an audio block inside one user turn. A Qwen2.5-Omni-3B encoder listens to the audio directly. It does not transcribe it first. AuK never needs a transcript of your reference clip, which is why voice cloning works on a recording in a language you did not tell it about. The encoder produces a representation of "this sound plus this request," and the diffusion transformer denoises against that.
The fusion between the two is not generic. The checkpoint ships its own learned Qwen layer-fusion weights, which is why the ComfyUI integration has to hand the AuK model to the encoding node rather than just running the text encoder standalone. It also means the model card's warning that "missing text_encoder.* keys during checkpoint loading are expected" is not a bug report. The encoder and VAE load from separate files at runtime by design.
Now the consequence. In a model with a task selector, the selector routes and the prompt decorates. Get the prompt slightly wrong and you still land in the right code path. In AuK there is no routing except the sentence. The encoder's reading of your words is the only thing that distinguishes "keep the singing voice" from "keep all human voices" from "keep only the speaker who says 'get what'." Three different tasks, three different outputs, one input field, and the only difference is English.
That is elegant. It is also fragile in a way the release notes do not mention.
Where the documentation stops agreeing with itself
The Cookbook publishes instruction templates for every task in English and Chinese, then gives a CLI example and a Python example for each. It is a good document. It is also, in at least five places, at odds with itself.
The worked examples do not match the templates they sit under. Speech separation's template reads "Keep only the {first/second/etc.} speaker to start talking and remove all other speakers." Its CLI example runs "Please keep the second speaker to start talking and remove the other speakers, outputting a single clean speech track." Music separation's template is "Keep only the singing voice and remove everything else." Its CLI example is "Keep the clean singing voice, drop all other audio." Instruct TTS has three renderings: the Cookbook template says "Generate speech based on the following description," the CLI example directly beneath it says "Based on the following description: '...', generate speech content '...'," and the main README's Python snippet uses a fully Chinese construction. Same task, three sentences, two of them on the same page.
If phrasing were incidental, none of that would matter. The people who tested the model say it is not incidental.
One example is running the wrong direction. Whisper conversion's CLI example loads wh-w2n-zh-input.wav, where w2n means whisper to normal, and applies the instruction "用小声耳语的方式把这段话说出来," which converts speech into a whisper. The main README uses that identical file in its Prompt Enhancer demo with the correct whisper-to-normal instruction. Two documents in the same repository, one input file, opposite intents.
Two of the shakiest tasks switch to Chinese in their worked examples without explaining why. De-accent publishes an English template, "Remove the regional accent while preserving the speaker's voice and content," and then demonstrates with a Chinese sentence that is not even the Chinese template. Whisper conversion does the same. Both tasks appear near the bottom of a reliability ranking that a third-party integrator published independently, and that ranking notes, without connecting it to the Cookbook, that "some unstable tasks succeed more often with the Chinese cookbook instruction than with its English version."
Nobody has put those two observations next to each other. I am putting them next to each other. The vendor's own worked examples switch language on exactly the tasks an outside tester found the English phrasing fails on, and the vendor never says why.
The Prompt Enhancer edits your sentence and drops clauses. The README shows PE taking "Convert this whisper into normal speech while preserving the speaker and content" and emitting "Convert this whispered speech into normal speech." The preservation clause, which every template in the Cookbook includes, is gone from the command PE tells you to run.
Two ComfyUI packs share a name and disagree on a hard limit. Tencent ships comfyui/ComfyUI-AuK inside its own repository. An independent developer ships Saganaki22/ComfyUI-AuK, created September 9 at 20:58 UTC with its converted weights following four hours later. They use the same node names. Tencent's ComfyUI guide states that "source/reference + generated target must fit within 30 seconds, after PE preprocessing, resampling, and model-frame rounding." The third-party pack's README states: "There is no combined source/target duration cap in this node pack." Both statements are probably true of their own code. Neither warns you the other exists.
Put this into practice
Start on the Space. It costs nothing, the demo inputs are already loaded, and it tells you within five minutes whether AuK does the specific job you need before you download six gigabytes.
When you move local, the ranking is your download decision. The prompt adherence list sorts eighteen tested behaviors from most to least reliable, and the top eight are description TTS, voice cloning, volume changes, speech enhancement, singing extraction, speaker separation, denoise-only, and quality repair. If your job is on that list, AuK is likely to be the cheapest tool you own for it. If your job is lyric editing, whisper conversion, accent removal or word replacement, read the bottom of the list before you plan a project around it.
For the ComfyUI route, install the node pack into custom_nodes/, install its requirements against the Python that runs ComfyUI (it wants Transformers 5.3.x), and pull three files from drbaph/AuK-comfyui: one diffusion model, one Qwen encoder, and the VAE. You do not need all twelve. Then drop 02_audio_edit_or_clone.json from example_workflows/ onto the canvas and change the instruction.
Three habits will save you most of the pain.
Treat the sentence as a parameter, not a request. Keep a text file of instructions that worked, verbatim, with the task and the seconds value beside each. When something fails, change one clause and rerun before you change the model. On a model where the sentence is the only routing mechanism, this is not superstition; it is the same discipline you already apply to a sampler setting.
Set the duration slightly short on hard edits. The tester's finding is specific: 0.2 to 0.5 seconds shorter than the source improved adherence on difficult edits, so 5.5 seconds for a 6-second input, even when the replacement text would normally need more room, not less. seconds=0 matches the source, and text-only generation needs a positive value.
Try the Chinese phrasing before you give up. Copy the CN template straight out of the Cookbook. This feels ridiculous if you do not read Chinese, and it is the single highest-yield trick available on the tasks that fail.
One chain deserves a warning of its own. Lyric editing requires an a cappella input, and the Cookbook tells you to get one by running Music Separation first. Music Separation is rank five on the stable list. Lyric editing is dead last. So the one workflow most people will want, pull the vocal out of a track and change a word, runs your best task directly into your worst one, and any artifact the first stage leaves behind arrives at the stage least able to cope with it.
What it cannot do, and what nobody has measured
The reliability gap is the headline limitation and I have covered it, but four more are worth naming before you commit an afternoon.
The hosted demo is not a clean control. Tencent's ComfyUI guide notes that its node "returns the generated waveform directly, without Gradio's extra loudness processing for lyric editing and vocal extraction; output levels may differ." Those are two of the tasks people will most want to A/B between the Space and a local install, and on those two the Space is doing something extra. The third-party tester states that comparisons against the demo used the same audio, instructions and PE path. That is careful work, and I still cannot rule out that a loudness difference sits inside those two comparisons. I have not measured it. Neither, as far as any published document shows, has anyone else.
The recommended path is not fully local. Tencent's Prompt Enhancer wants an OpenAI-compatible LLM, and the README's example configuration points LLM_BASE_URL at Tencent Cloud with LLM_MODEL_NAME="hy3", with optional Tencent Cloud speech recognition on top and a local SenseVoiceSmall fallback for ASR. So the officially recommended way to turn a free-form request into a model instruction sends your request to a cloud endpoint. The third-party pack reimplements PE locally using the Qwen language head and says outright that "classification quality can therefore differ from upstream's recommended hy3." Nobody has published how much.
The MIT license does not cover the whole stack. AuK's weights and code are MIT and the LICENSE file says so. The encoder is Qwen2.5-Omni-3B, and the third-party pack's attribution states that the bundled Qwen weights "retain the upstream Qwen Research license" and that conversion does not relicense them. If your use is commercial, that is the term to read, not AuK's.
And nobody has published a quality comparison between Base and Flash on editing accuracy. The pack says Base at 32 steps is the quality choice and Flash at four steps is for speed, which is the normal shape of a distillation claim. On a model where the failures are about instruction understanding rather than sample quality, the interesting question is whether four steps makes the shaky tasks shakier. That measurement does not exist in any document I could find.
The part worth keeping
The thing I keep coming back to is not the model. It is that a stranger with twenty-five GitHub stars wrote down which eighteen behaviors work and put the warning above the download links, in a paragraph beginning "Read this before installing." He gained nothing by doing that. It cannot have helped his download count.
It is also, right now, the most useful document in the AuK ecosystem, and it is not written by the lab.
That tells you something about where the real documentation for open models is going to come from. The lab publishes the templates and the benchmark chart. The person who spent a weekend wiring it into the tool you actually use publishes the part that changes your afternoon. Both are necessary and only one of them is currently reliable.
So read the model card, then find the integrator's README, and when the two disagree, believe the one that makes the model look worse. Then write your own list. Your tasks are not the tester's tasks, your audio is not his audio, and the ranking that matters is the one you build from six runs on your own material. AuK is small enough and free enough that building it costs you an evening.
Start with the two tasks you would pay money for. If they are in the top eight, you have a new tool. If they are in the bottom ten, you have saved yourself a week and learned exactly which sentence to try next.
Medium metadata
- Title: AuK Makes the Sentence the API, and Its Own Docs Cannot Keep the Sentences Straight
- Subtitle: A 1.5B MIT-licensed speech model does sixteen jobs through one text box. That design puts unusual weight on getting the words right, and the words in the documentation disagree with each other.
- Tags: AI Audio, Open Source, Speech Synthesis, Machine Learning, Creative Tools
- Suggested reading time: 10 minutes
- Canonical subject: Tencent Hunyuan AuK (arXiv:2609.08936), MIT license, ComfyUI and MLX ports, September 2026