Your Local Music Model Is Not Broken. The Wrapper Around It Is.
A song generator that produced noise on a Mac turned out to be fine. The message being handed to it was in the wrong shape, and nothing anywhere said so.
Somebody spent last week trying to make an open music model sing on a MacBook, got noise, and did the unusual thing: instead of shrugging and calling the model bad, he measured exactly how bad and then found out why.
The answer is worth your attention even if you never touch this particular model. The weights were fine. The conversion was fine. What was wrong was the format of the message being handed to the model, and the failure mode was not an error. It was a plausible-sounding, completely useless output that any reasonable person would have blamed on the model.
That failure shape is common now, and almost nobody writes about it.
The setup
ACE-Step 1.5 generates songs with sung lyrics from a style description. You give it something like "upbeat pop song with female vocals, bright synths, driving beat" plus your actual words, and you get back 48 kHz stereo audio with those words sung. The ACE-Step team released it in January under the MIT licence. The person who converted it for Mac describes the original team's training data as licensed and royalty-free, which if that holds up makes this one of the very few music generators you could put in paid work without a lawyer in the room.
Last night a conversion of it for Apple Silicon appeared: roman220220/ACE-Step1.5-sft-MLX-8bit, created at 19:30 UTC on Friday and last touched at 06:25 this morning, alongside a full-quality version and a smaller one for 16 GB machines. Also MIT. About 5.6 GB to download in total, roughly 49 seconds of compute for 30 seconds of music on an M5, peaking around 8.6 GB of memory.
Run it the obvious way and it produces cacophony. No music, no words.
What went wrong
There are two versions of this model. One of them expects to be handed a rough plan first, a sketch of the song written by a separate small model that decides the shape before any audio gets made. The other one does not. It writes the song itself.
The toolkit that everyone uses to run audio models on a Mac, mlx-audio, was built around the first version. So it dutifully produces the plan and hands it over. The second version, the one that sounds better and sings more clearly, was never trained to receive that. It chokes on it, and what comes out the other end is noise.
Nothing errors. Nothing warns. You get a file, you play it, and you think: this model is bad.
Turn the planner off and it sings. It is also about fifteen seconds faster per track, because you are no longer running a model whose only contribution was to break the next one.
There is a second, smaller version of the same problem. When these models are steered toward your prompt, they run twice internally, once with your description and once with nothing, and the difference between the two results is what pushes the output toward what you asked for. That "nothing" is supposed to be a specific blank value the model was trained alongside. The toolkit was sending plain zeros instead. Zeros are not nothing. They are a real input the model has never seen, and it drags the result sideways. Measurably: lyric clarity went from 0.28 to 0.22 on his test set once the trained blank was used instead, about a fifth better.
How he proved it, which is the part to steal
The reason this writeup is worth reading rather than just following is the method.
Judging a music generator by ear does not scale. You need a lot of tracks to tell signal from seed luck, and after eight of them you cannot hear straight anymore. So he built a number: run the generated audio through a speech transcriber, compare the words it heard against the lyrics he asked for, and score the gap. Lower is better. It measures one thing only, whether the words come through, and he is explicit about that.
Then he checked the metric itself, which is the bit most people skip. He had a computer voice read the same lyrics over an instrumental at full volume and at half, transcribed that, and got a perfect score both times. So music does not hide intelligible words from the transcriber. When the score is bad, the singing is bad, not the measuring.
With that in place the diagnosis becomes arithmetic, and it is worth being careful about which numbers pair up. On the other version of this model, the one the toolkit was actually built for, the toolkit's default path scored 0.94, meaning almost nothing was intelligible, while the same weights through the official pipeline scored 0.66. Bad either way, but a third of the gap was the wrapper rather than the model.
On the version this article is about, the comparison is starker and less tidy, because there is no score for the broken path. It makes cacophony. Once the planner was switched off, the first test track came back at 0.07, and the full run settled at 0.22 against the official pipeline's 0.17. Practically the same, three times faster, on a Mac.
I have not found a cleaner demonstration that "the model is bad" and "my setup is wrong" are two claims most of us never bother to separate.
Put this into practice
If you want songs with audible lyrics running locally on a Mac, here is the whole path.
Install the toolkit at a pinned version. mlx-audio from the pc/add-ace branch at commit 1e8264a. Pin it. Every measurement in this article was taken against that commit, and the branch is active.
Pick your build by memory, not by instinct. The full-quality version peaks around 10.6 GB and takes about 36 seconds per 30-second track. The 8-bit build linked above uses about 8.6 GB and takes about 49 seconds. The 4-bit build fits a 16 GB Mac at about 7.5 GB. Counterintuitively, 8-bit is the slowest of the three on this hardware, so if you have the memory, take the full-quality one and get both better output and better speed.
Apply the two fixes. The model card has them as a code block of about twenty lines. Pass use_lm=False to switch off the planner. And wrap the part that reads your prompt so that when it gets handed an empty description, it substitutes the model's own trained blank value instead of zeros. Both are copy-paste.
Generate a couple of seeds. This is the author's advice and the measurements back it. On the same song with the same settings, scores ranged from 0.1 to 1.0 across different random starts. That spread is larger than the difference between any two of the builds. One bad track tells you nothing.
Settings that were actually used: 50 steps, guidance 7, and the shaped-guidance option the official pipeline uses. Those are in the card's example call.
Where it breaks
The score measures intelligibility and nothing else. It does not know whether the mix is any good, whether the timbre is right, or whether the song is boring. The author is direct about this, and about something that complicates his own conclusion: the faster variant, the one that scores much worse on lyrics, sounds fuller and more like a finished record. He prefers the clear-lyrics one on electronic music and says so as a preference, not a ranking. Two different trades, not a winner.
The sample is small. Three songs, four random starts each, twelve tracks. He says so repeatedly and warns that a few points of difference is noise. Do not read the tables as precise.
Every timing is one person on one M5 with 26 GB. Your machine will differ. The memory peaks are the numbers most likely to bite you, and they scale with how long a track you ask for, so a 60-second request will not behave like the 30-second measurements.
There is a real risk in pinning a commit, too. When that branch merges or moves, these fixes may be unnecessary, or they may break. The card is a snapshot of a moving target, and nothing will tell you when it goes stale.
And the licensing claim deserves the care it does not usually get. MIT on the conversion and MIT on the base model are both verifiable on the repository pages. The claim about licensed and royalty-free training data is the converter's one-line description of the original team's data, not something the original team is quoted saying. That is worth checking against the original team's own statements before you build a commercial release on it.
What this is really about
The specific fix will be obsolete within weeks. Somebody will merge it upstream and this whole article becomes a footnote.
The pattern will not be obsolete. Every good open model now reaches you through two or three layers of other people's conversion work, and the most fragile joint in that chain is not the maths. It is the shape of the message being passed. The weights get checked obsessively, with error percentages published to two decimal places. The prompt format gets checked by nobody, because it is not glamorous and it does not produce a number.
So when a local model disappoints you, the useful question is not whether the compression hurt it. It is whether the thing feeding it is speaking the same dialect the model was trained on. That question is cheap to ask and it has, in this case, been the difference between noise and a song.
The next time a well-reviewed open model sounds terrible on your machine, spend twenty minutes comparing what your toolkit sends against what the original pipeline sends. You may find the model was always fine. Or you may confirm it really is bad, which is also worth knowing, and worth writing down the way this person did.