This Kokoro Core ML Conversion Hands You a Duration for Every Phoneme. That Is the Whole Subtitle Problem, Solved for Free.

How splitting a text-to-speech model in two turns word-accurate caption timing into a byproduct of synthesis, why the default compute placement makes it 5.6 times slower, and the calibration number that tells you when a conversion is broken.

Every text-to-speech model on your machine already knows exactly when each syllable starts. It has to. Something in there decided how many frames of audio the "str" in "strange" gets before it could render a single sample. Then it throws that away and hands you a WAV file, and you go find a forced aligner to work out what it just told you and forgot.

A Core ML conversion of Kokoro-82M published on September 6 stops throwing it away. It returns a duration for every phoneme alongside the audio, from the same forward pass, because of where the author chose to cut the model in half. The card puts the consequence plainly: that is the timing AVSpeechSynthesizer declares and does not report.

If you make videos, courses, audiobooks, or anything where a caption has to land on a word, that sentence is the whole article. The rest is how it works and what it costs.

The stakes: alignment is a second pipeline you should not need

Right now the normal way to caption synthetic narration is to generate audio, then run it back through Whisper or an aligner like Montreal Forced Aligner, then reconcile the transcript against the script you already had. You are asking a second model to reverse-engineer information the first model computed and discarded. That pass costs time, adds a dependency, introduces its own errors on proper nouns and numbers, and produces word boundaries that are estimates.

The alternative is not exotic. It is just a matter of where you split the graph.

The mechanism: cut at the place where the shape stops being knowable

Kokoro is a StyleTTS2 descendant, 82 million parameters, Apache 2.0, and it has been the default small local voice for a while now, sitting at 6,823 likes and past 11.4 million downloads since December 2024.

Internally it does something that makes Core ML unhappy. It predicts a duration in frames for every phoneme, then repeats that phoneme's feature vector by however many frames it just predicted, then renders audio from the repeated sequence. Every tensor shape after that step depends on a number the model computed at runtime. ONNX papers over this with an axis whose size is resolved at run time. Core ML has no equivalent.

The conversion splits the model at exactly that seam, into two packages:

  tokens, style ──▶ KokoroProsody ──▶ prosody, text, duration
                                            │
                    gather (caller) ◀───────┘   repeat column i, duration[i] times
                            │
  prosody', text', style ──▶ KokoroAcoustic ──▶ audio

The step that moves out to your code is an index gather. All the arithmetic stays inside the graphs. And because the split falls there, duration becomes an output you can read rather than an internal tensor you never see. It comes back in frames, unrounded and unscaled, and one frame is 600 samples at 24 kHz, so 40 frames per second. Round each to at least 1, sum them, and you have a start time for every phoneme in the utterance.

That is the trick. A constraint in Apple's runtime forced a split, and the split happened to expose the most useful intermediate value in the model.

The interfaces are small enough to hold in your head. KokoroProsody takes tokens at [1, 3…512] int32 and style at [1, 256] fp32, and returns prosody [1, 640, T], text [1, 512, T] and duration [1, T]. KokoroAcoustic takes the gathered prosody and text at [1, 640, F] and [1, 512, F] plus the same style vector, and returns audio at [1, F·600].

Two more design choices are worth naming because they are the difference between a demo and something you would ship.

Variable-length inputs, no buckets. Tokens accept anything from 3 to 512, which is the model's own phoneme context limit, so there is no padding to fixed sizes and no cap on utterance length below the model's real ceiling. Compare that to mattmireles/kokoro-coreml, the other Core ML conversion in circulation, which this card names directly as an alternative. Its file listing tells the story: decoder packages compiled at 3s, 7s, 10s, 15s and 30s, and duration models at t32, t64, t128, t256, t320, t384 and t512. That is the bucketed approach, and it is a reasonable trade rather than a mistake. Buckets keep shapes static, which is what the Neural Engine wants.

Voices as a memory-mappable blob. Voices.bin is 28.2 MB and holds all 54 published voices as 510 style vectors each, one per phoneme count, 256 numbers per vector, with an eight-byte AIOSVOX magic and a documented header. You pick the row matching your phoneme count and pass it in. No unpickling, no per-voice file juggling, and you can mmap the whole thing.

The number that tells you a conversion is lying

Here is the part I want more model cards to copy.

The card reports spectral correlation of 0.977 against the PyTorch reference on a held-out utterance, and immediately explains why waveform correlation would be meaningless: Kokoro's vocoder adds random noise to its unvoiced excitation, so two runs of the reference do not match sample for sample either.

Then it gives you two calibration points. Two reference runs, differing only in that noise, score 0.998. And a conversion carrying a known modulo bug scores 0.801 while producing audio of the right length, the right envelope and the right word timing.

Sit with that second one. A broken conversion of this model does not crackle, cut out, or produce obvious garbage. It produces audio that passes every structural check you would think to run and sounds like a slightly worse voice. The only thing that catches it is a spectrum comparison, and now you know the number to compare against. The card's advice is one line: if you convert this model yourself, check spectra.

That is the same failure shape the four-bit quantization work has been running into all week, and it is becoming the defining problem of the porting era. Conversions do not fail loudly. They fail plausibly.

Put this into practice

Two useful things to build here. Start with the second one if you only want the timing.

Getting the model running.

1. Pull the three files from toddkrabach/Kokoro-82M-CoreML: KokoroProsody.mlpackage.zip at 36.0 MB, KokoroAcoustic.mlpackage.zip at 114.3 MB, and Voices.bin at 28.2 MB. Under 180 MB total. You need macOS 15 or iOS 18 and up.

2. Set compute units per model, and do it before you benchmark anything. This is the single highest-value line in the card. On an M4 Pro rendering three seconds of audio, KokoroProsody takes 5.8 ms on .cpuOnly and 32.3 ms on .all, while KokoroAcoustic takes 58.0 ms on .cpuOnly and 41.5 ms on .all. Run each on its best unit and you are at about 47 ms total, roughly 63 times realtime. Accept the defaults and the prosody half runs 5.6 times slower than the CPU alone would do it, because it is a small BERT-and-LSTM graph over a few dozen tokens and shipping that work to another compute unit costs more than the work. Neither package touches the Neural Engine at all, because variable shapes keep them off it. Loading and compiling both packages is a one-time 0.35 seconds.

3. Feed it phonemes. Tokens are the Kokoro phoneme vocabulary with token 0 at both ends. Look up the Voices.bin row for your phoneme count, excluding those two boundary tokens.

Getting timings out, which is the actual point.

4. Read duration from the prosody pass, before you render. It comes back per phoneme in frames. Round each up to at least 1. If you want a speed change, divide here rather than re-running the model.

5. Convert to seconds and accumulate. 40 frames per second, so phoneme i starts at sum(durations[0..i-1]) / 40 and ends at sum(durations[0..i]) / 40. Note the exclusive upper bound on the start time; summing through i gives you the end, and that off-by-one is the easiest mistake to make here. The formula is mine, derived from the card's frame size and sample rate, not something the card states.

6. Roll phonemes up to words. You know which phonemes came from which word because you built the token sequence. Sum the durations inside each word's phoneme span and you have word start and end times, which is what SRT, VTT and every animation timeline actually want. No aligner, no second model, no reconciliation step.

7. Handle the words your dictionary does not have. The repo ships a converted grapheme-to-phoneme model, G2PEncoder and G2PDecoder, together 2.8 MB, a one-layer BART trained on the misaki dictionaries so it emits the same alphabet the voice reads. It covers roughly 1.5% of ordinary text, mostly proper nouns and technical terms. Both graphs are fixed at 32 tokens and padded. Feed the encoder bos + graphemes + eos, start the decoder at decoder_start_token_id, read logits[step], write the argmax to position step + 1, stop at eos. There is no key-value cache and the card explains why: re-running a one-layer 128-wide decoder is cheaper than keeping one.

The trap in step seven, stated before it gets you. The repo ships g2p-vocab.json with a graphemeChars list for the input side. Do not decode the output through it. Input and output share one embedding and agree only up to id 25; above that, one alphabet is spelling and the other is phonemes. Decode the wrong way and "loosen" comes back as lTusXn, which the card describes as real phonemes wearing the wrong letters. It looks exactly like a broken model and is not one. Read the output through the phoneme table.

Honest limitations

English only, and only as good as Kokoro. This is a conversion, not an improvement. The NOTICE lists seven conversion changes and states plainly that no weights were retrained, fine-tuned or altered. Kokoro-82M is a small StyleTTS2 model and it sounds like one. It will not do the expressive range of a multi-billion-parameter voice model, and no amount of good engineering around it changes that.

Not on the Neural Engine, and the GPU is mandatory. Variable shapes are what buy you the unbucketed lengths and they are also what keep both packages off the ANE. The card also says neither package runs with the GPU excluded. On a machine with a weak GPU, the bucketed mattmireles/kokoro-coreml conversion may genuinely do better, and the card says so itself rather than pretending its approach dominates. Measure on your target hardware.

The phonemizer is a fallback, not a solution. Scored against the dictionary it was trained on, it gets 63% of words exactly right with a 9% phoneme error rate, matching the PyTorch original it came from. That is fine for the 1.5% of text that needs it and would be a disaster as a primary path. Proper nouns are where you will feel it, and proper nouns are exactly what a narration script is full of.

One fidelity measurement, one utterance. The 0.977 figure is a held-out utterance, singular. The calibration points around it are the valuable part and they are also single measurements. Nobody has run a broad perceptual evaluation of this conversion.

Per-phoneme durations are the model's prediction, not ground truth. They tell you when the model intended each phoneme to occur, which is exactly right for captioning synthetic speech and tells you nothing about recorded speech. If you are aligning a human voiceover, you still need an aligner.

The vectors are somebody else's. The 54 voices are redistributed from the model-files-v1.0 release of thewh1teagle/kokoro-onnx, which sits at 2.7k stars. Apache 2.0 all the way down, and the attribution is done properly, but know where your voices came from.

Where this leaves you

The reason to care about this conversion is not that it is fast, though 63 times realtime on a laptop is a good number. It is that a runtime limitation forced a split at the one seam in the model where the useful intermediate value lives, and the author noticed and exposed it instead of hiding it back inside a wrapper.

That is worth generalizing. Most TTS architectures with an explicit duration predictor have the same seam. StyleTTS2 has it, FastSpeech-family models have it, and anything with a length regulator has it by construction. If you are building narration tooling on any of them, the timing you are paying an aligner to recover is probably sitting one tensor away from where you already are.

The thing I would fix first is smaller and more annoying. Right now the per-phoneme durations are a Core ML output you have to plumb yourself into an SRT writer or a Premiere marker file or a Rive timeline. Nobody has shipped that last mile. It is maybe two hundred lines and it would turn a good conversion into a workflow.

If you build it, or if you run the spectral check on your own conversion and get something other than 0.977, that is the result I want to hear about.

---

Medium metadata

Recommended reading time: 8 minutes

Primary sources:

• The conversion: https://huggingface.co/toddkrabach/Kokoro-82M-CoreML
• Upstream model: https://huggingface.co/hexgrad/Kokoro-82M
• The bucketed alternative named in the card: https://huggingface.co/mattmireles/kokoro-coreml
• Grapheme-to-phoneme source model: https://huggingface.co/PeterReid/graphemes_to_phonemes_en_us
• Voice vector provenance: https://github.com/thewh1teagle/kokoro-onnx