Shrink Your LoRA Adapters by 80 Percent With an SVD Rank Resize
The technique that takes a 1,866 MiB adapter down to 312 MiB, the command that does it, and why the impressive-looking fidelity number on the model card is not measuring what you think it is.
A speed adapter for MiniMax H3 ships at 1,865.57 MiB. Someone ran a truncated SVD across it and published a 311.79 MiB version, an 83.29 percent reduction, reporting a cosine similarity of 99.9190 percent to the original weight update.
Sitting underneath those numbers on the same model card is a sentence most people skip: the figures are "numerical weight-reconstruction measurements, not a perceptual video-quality score."
Both halves of that are worth your evening. The technique is real, it is ninety years old, it is boring linear algebra, and you can run it tonight on every adapter on your drive. The measurement is real too, and it is not the measurement you would choose if you actually wanted to know whether the small version still looks right.
Why this is worth an evening
Most people training LoRAs pick a rank at the start and live with it. Rank 128 feels safe, so rank 128 it is, and the file comes out at whatever size it comes out at.
That is backwards. The advice from the contributor who wrote the resize tool, in the pull request that landed it, is to train at the highest rank you can and then resize down: "one is better to create a LoRA in two steps. Use the highest rank possible... then resize it to a much smaller size with minimal loss. This provide way better results vs trying to train a small rank LoRA in one step."
The reasoning holds up on its own. Rank during training is a capacity constraint on what the adapter can learn. Rank after training is a storage question about what it already learned. A high-rank run explores a bigger space and lands somewhere better. Once it has landed, most of the singular values in that solution sit near zero, costing you disk, VRAM, and load time for nothing.
The tool's own test set was three character LoRAs pulled from CivitAI, resized at a ratio threshold of 5. One of them went from 295MB to 19.4MB once a clamping step was removed from the pipeline.
If you keep thirty character adapters and swap them constantly, this is the difference between a folder that stays in cache and one that does not.
How it actually works
A LoRA does not store a modified weight matrix. It stores two thin matrices whose product is the change you want to apply. The effective update is ΔW = B @ A (in kohya's naming, lora_up @ lora_down), and the rank is the shared inner dimension.
Three moves:
- Reconstruct ΔW from its factors.
- Take the singular value decomposition, giving you U, S, and Vᵀ.
- Keep the top r singular values, discard the rest, and fold the survivors back into new A and B matrices.
The reason this is principled rather than a hack is the Eckart-Young theorem, published in Psychometrika in 1936. A truncated SVD gives you provably the closest rank-r approximation to the original matrix in Frobenius norm. There is no better rank-r answer to go looking for.
One caveat on that, because it changed recently and the default behavior surprised me. Current sd-scripts uses torch.svd_lowrank by default for large matrices, which is a randomized approximate SVD, not the exact one. The optimality guarantee applies to the exact decomposition. If you want it, pass --svd_lowrank_niter 0 to force the full torch.linalg.svd path and accept that it is slower.
The trick the big video-model repacks use avoids materializing dense ΔW at all: QR-decompose B and Aᵀ, form the small r-by-r matrix M = R_B R_Aᵀ, SVD that tiny thing, and recover the factors from the Q matrices. The singular values of M equal those of BA, so you get the exact answer without ever building the big matrix.
Per-layer rank is where it gets interesting
The naive version picks one rank for the whole adapter. The good version picks a rank per layer.
Look at what the H3 repack produced: rank sum 6,778 across 312 independently ranked projections, an average of 21.72, with individual projection ranks ranging from 2 up to 105. Some layers needed a hundred directions. Some needed two. A single global rank would have overpaid for the second group and starved the first.
The filename says "rank 21" because that is the floor of the average, not because any particular layer was assigned 21.
Put this into practice
The tool is networks/resize_lora.py in kohya-ss/sd-scripts. If you have trained a LoRA locally you probably already have this repository.
One trap before you start: do not run the copy at tools/resize_lora.py in the bmaltais/kohya_ss GUI fork. That file is a stale pre-2023 version with no --dynamic_method at all, and the command below will fail on it with unrecognized arguments. The GUI itself shells out to sd-scripts/networks/resize_lora.py, which is the file you want.
Start with one adapter you know well, so you can actually judge the output.
python networks/resize_lora.py \
--model input_lora.safetensors \
--save_to resized_lora.safetensors \
--new_rank 32 \
--dynamic_method sv_fro \
--dynamic_param 0.9 \
--save_precision fp16 \
--device cuda \
--verbose
Four flags do the real work.
--dynamic_method picks how each layer's rank gets chosen. sv_fro keeps enough singular values that the cumulative sum of their squares reaches dynamic_param² of the total. sv_cumulative does the same on the plain sum. sv_ratio keeps every singular value above S[0] / dynamic_param, a relative-magnitude cutoff rather than a budget.
--dynamic_param is the threshold, and its valid range depends on the method. sv_ratio wants a value greater than 1; the kohya GUI enforces 2 as a hard floor, though the script itself does not check. sv_fro and sv_cumulative both live between 0 and 1, and sv_fro should sit considerably higher than you would set sv_cumulative.
--new_rank changes meaning once --dynamic_method is set. The help text is explicit: it becomes "a hard limit for max rank" on any single layer rather than the target.
--verbose prints per-layer retention, and you should read it. With a per-layer method it shows the chosen dim: and alpha: for each layer, then closes with an average Frobenius norm retention and a standard deviation.
Two details that matter and are not in the flag list: the merge math is forced to float32 internally regardless of --save_precision, and with a per-layer method the saved metadata sets ss_network_dim and ss_network_alpha to the literal string 'Dynamic'.
Picking a method
Use sv_fro when you want the smallest file that still reproduces the adapter faithfully. It optimizes for reconstruction accuracy, which is what you want for an adapter you already like.
Use sv_ratio when the adapter is overfit and you want the resize to also denoise. The tool's author is explicit that sv_fro is "not a good option for users [trying] to salvage an overfit LoRA," and the logic is clean: faithfully reproducing an overfit adapter faithfully reproduces the overfitting. sv_ratio discards small directions regardless of energy, and small directions are where memorized noise tends to live.
Expect to iterate. Singular-value distributions differ per adapter and there is no universal threshold. The H3 repack put it the same way: "A separate Frobenius threshold was solved for each LoRA because their singular-value distributions differ."
If you are on SDXL
elias-gaeros/resize_lora, MIT licensed, scores directions against the base checkpoint rather than the adapter alone. It also takes a file-size budget directly, which is a friendlier interface than tuning a threshold:
python resize_lora.py sdxl_base.safetensors my_lora.safetensors -o out \
-r "fro_ckpt=1,thr=-3.5"
Swap thr= for size=48 to say "give me the best 48 MiB version" and let it do greedy score-per-byte selection. Its -vv output prints per-layer lines like dim: 8->5 rle_lora: 3.19% rle_ckpt: 0.03%, giving you two relative errors: one against the adapter's own norm, one against the base layer's. SDXL and LoCon only.
Honest limitations
Per-layer rank broke at least one popular loader. The documented case is specific rather than general: dynamically resized adapters threw size-mismatch warnings in sd-webui-additional-networks while working fine with the webui's built-in LoRA support, and the extension needed updating. The H3 repack's validation section explicitly confirms "ComfyUI recognizes all 208 adapters and all 416 tensors," and that check exists because this is the class of thing that fails. Test in your actual loader before deleting the original.
Whole layers can disappear. The elias-gaeros tool prints dim: 256->0 when it decides a layer contributed nothing, reporting 100 percent relative error for it. Sometimes correct. Sometimes it means your threshold just deleted something you needed.
There is a clamping default still shipping that degrades output. CLAMP_QUANTILE = 0.99 was found to hurt quality; setting it to 1 produced results much closer to the original. It was removed from resize_lora.py, but extract_lora_from_models.py on current main still exposes --clamp_quantile with a default of 0.99, and there is an open issue against a downstream ComfyUI trainer node about exactly this. If your workflow extracts a LoRA from two checkpoints before resizing, pass --clamp_quantile 1.0.
The headline fidelity metric is not perceptual, and the two numbers on the card are one number. Cosine similarity of the reconstructed ΔW is a weight-space measurement. It says the matrices are close. It does not say the video looks right, and the author says so himself. Worse, when a card prints cosine similarity and relative L2 error side by side as if they were independent evidence, they are not. Because a truncated SVD is an orthogonal projection of ΔW, the residual is orthogonal to what you kept, so ‖ΔW − ΔŴ‖² = ‖ΔW‖² − ‖ΔŴ‖², which collapses to rel L2 = √(1 − cos²). Check it against the published figures: 99.9190 percent cosine predicts 4.0241 percent, the card reports 4.0234. On the rank-28 file, 97.3279 percent predicts 22.9625 and the card reports 22.9625 exactly. One measurement, printed twice.
Cosine of ΔW is also not the field's usual metric. The kohya script reports Frobenius norm retention (a norm ratio, not an energy ratio). elias-gaeros reports relative errors. The Delta-SVD preprint reports CLIP, SSIM, and FID. It is a defensible choice; it is not a standard.
Fidelity varies wildly at similar compression. In that same repack, the Ref2V file holds 4.02 percent relative L2 error while a rank-28 file sits at 22.96 percent with minimum projection Frobenius retention down to 92.67 percent. Those are two different source adapters, not the same one squeezed twice, and that is the point: the outcome is set by how the singular values happened to be distributed in whatever you started with. You cannot pick a threshold once and reuse it.
One thing I could not verify and will not guess at. Whether rank-resized adapters degrade worse than full-rank ones when you stack several at once. No primary source measures it, adjacent reports do not isolate rank as the variable, and I am not going to assert it either way. Same for whether a quantized base model changes anything. If you have data, it would be genuinely useful.
A licensing landmine specific to the H3 example. The repack I have been quoting carries an apache-2.0 tag, as does the upstream turbo adapter it derives from. The MiniMax H3 Community License defines "Excluded Territories" as the European Union, the United Kingdom, the Republic of Korea, and the United States of America, and section V.4 prohibits using, reproducing, modifying, distributing, or displaying the works "or any of their Outputs or results" outside the Applicable Territory. That reaches the outputs, which is broader than a deployment ban. The hosted API is not restricted the same way, and there is a formal application path for organizations in excluded territories.
I am not asserting anyone violated anything. I am pointing out that a parallel repack of essentially the same lineage publishes under minimax-h3-community-license-agreement rather than Apache 2.0, which is direct evidence that the tag is a choice rather than a settled fact, and that a US or EU reader downloading on the strength of an Apache label is relying on something the base-model license appears to contradict.
None of that touches the technique. Rank resizing is license-neutral. It is the specific adapter in my headline example that carries the problem.
Try it on something you can judge
Pick one adapter whose failure modes you already know. Resize at sv_fro 0.9, again at 0.8, generate the same seed through all three, and look.
What you are hunting for is the point where the number still looks fine and the output stops being fine. That gap is the whole story. Frobenius retention of 96 percent sounds like nothing was lost. Whether the face still looks like the face is a different question and the only one that pays.
Public data on this is thin but not nonexistent. There is at least one write-up that resized four SDXL character LoRAs across three thresholds and published the image grids next to the file sizes, and it is the most useful artifact of its kind I found. What does not exist is anything equivalent for video models, or for the kohya dynamic_param values specifically, which is where most people are working now.
If you build that table for a model family, publish it. Everyone doing this is currently rediscovering the same thresholds alone.
Medium metadata
Primary sources
- kohya-ss/sd-scripts,
networks/resize_lora.py: https://github.com/kohya-ss/sd-scripts/blob/main/networks/resize_lora.py - The pull request that added per-layer rank selection: https://github.com/kohya-ss/sd-scripts/pull/243
extract_lora_from_models.py, where the 0.99 clamp default still lives: https://github.com/kohya-ss/sd-scripts/blob/main/networks/extract_lora_from_models.py- elias-gaeros/resize_lora (MIT, SDXL/LoCon, base-aware scoring): https://github.com/elias-gaeros/resize_lora
- The H3 per-layer-rank repack and its validation table: https://huggingface.co/drbaph/MiniMax-H3-Turbo-Lora-ComfyUI
- A parallel repack published under the H3 community license instead: https://huggingface.co/Abiray/MiniMax-H3-Turbo-Lora-Pruned-ComfyUI
- MiniMax H3 Community License: https://huggingface.co/MiniMaxAI/MiniMax-H3/raw/main/LICENSE
- MiniMax license Q&A: https://huggingface.co/MiniMaxAI/MiniMax-H3/blob/main/docs/QA-about-License.md
- Delta-SVD preprint, the research version of the same idea: https://arxiv.org/abs/2508.16863
- A published threshold-to-image comparison for SDXL: https://civitai.com/articles/5381/resizing-sdxl-loras-in-seconds-instead-of-minutes
- ComfyUI: https://github.com/comfyanonymous/ComfyUI