You Can Now Direct an AI Video Model With a Drawing
MiniMax H3's new control pack adds scribble, box-layout and grayscale input. That sounds like a minor version bump. It changes who the tool is for.
Every way of steering an AI video model until now has had the same hidden requirement: you needed footage first.
Pose control needs a person to track. Depth control needs a scene to measure. Edge control needs a picture to trace. All of them are ways of saying "make something new that moves the way this existing thing moves," and every one of them starts with a camera or a render. Useful, but it makes the tool a post-production instrument. You bring it something finished and it gives you a variant.
On Tuesday, September 22, Alibaba's PAI group posted version 2.0 of its control pack for MiniMax H3, the open-weights video model. The write-ups have led on the count: eight input types now instead of five. That framing misses what actually happened. Three of the eight are new, and all three of the new ones accept things you can make on paper.
You can hand it a scribble, meaning rough hand-drawn lines. You can hand it a layout, meaning coloured rectangles on a white background that say where each subject sits in the frame. You can hand it a grayscale pass, meaning the light and shadow of a shot with no colour or detail in it.
None of those require a camera. Two of them you could draw at a kitchen table.
Why the three new ones are a different kind of thing
Look at what each of the original five carries.
Canny and HED carry outlines. MLSD carries straight lines, which is architecture. Depth carries the shape of space. Pose carries a skeleton. Every one is a measurement extracted from something real, and you get it by running a detector over footage you already have.
Scribble carries intent. Layout carries staging. Gray carries exposure. Those aren't measurements of an existing shot; they're decisions a director makes before anything is shot at all.
That is the whole story, and I have not seen anyone say it out loud. The control pack stopped being a way to remix your footage and started being a way to draft a shot. You block the frame with boxes, you sketch the gesture, you set the light with a gray pass, and the model fills in what the thing looks like. That is closer to how a storyboard works than to how a filter works.
The gray one deserves its own sentence, because it's the sleeper of the three. Feed it the luminance of a shot you like and everything about how that shot is lit carries over while the content is replaced entirely. If you have a grade you keep going back to, you now have a way to make new footage inherit it.
How the thing actually works, in plain terms
The download is not a video model. That matters for anyone budgeting a machine for it.
It's an add-on file that attaches to MiniMax H3 and has to be loaded on top of it. The base model brings the picture-making; the add-on brings the steering. The file holds only the steering part, about 13.5 GB.
The way it steers is worth one sentence, because it explains the one dial you'll actually touch. Your control clip gets read at ten separate points as the model builds the picture, and at each point it nudges the result toward your input. Version 1 read it at five points. Doubling the attachment points is what the authors describe as tighter adherence to your control clip, and it's also why the file roughly doubled in size, from about 6.8 GB to about 13.5 GB.
The dial is called control_context_scale. At 1.0 the model follows your drawing hard. Lower numbers let it drift. At 0.0 it ignores your input completely. Every published sample was made at 1.0.
There's also a proper masked-region redraw in this version. You supply your shot plus a second clip that's white where you want new content and black where the original must survive, and the model repaints only the white part. The authors changed how the empty region is handled between version 1 and 2.0, and they say the new recipe makes repainted areas blend into the kept areas more cleanly. That's their own report, not a measured result, but the reasoning they publish is specific enough to be worth testing.
What it costs to run
Here's where the enthusiasm should get careful.
The add-on file is 13.5 GB. That's the small part. The model card states that the base model runs about 62 GB and the piece that reads your prompt runs about another 62 GB, and that the two of them will not fit on a single 80 GB card fully loaded. You run it with the pieces shuttling on and off the card, which the project supports and which costs you time.
So the honest cost picture is: this is a rented-hardware job, not a laptop job. If you already run MiniMax H3 on rented cards, the add-on is a modest extra download. If you don't, the control pack is not your entry point.
There is no ComfyUI loader for it. Everything goes through the project's own code, VideoX-Fun, by editing variables at the top of a script and running it. That's a real barrier for anyone whose entire practice lives in a node graph. I expect that to change, because it changed for every previous release of this shape, but today it hasn't.
Put this into practice
The fastest useful thing you can do with this, if you have access to a machine that will run it:
Start with gray, not with scribble. Take a thirty-second clip with a look you already like. Convert it to grayscale with any editor you own. That's your control clip. Write a prompt describing a completely different subject and scene. Render. What comes back should be new content carrying your original's light. This is the cheapest test in the set because making the control clip takes one export and no drawing skill, and it tells you immediately how hard the model follows.
Then try layout. Make a short clip that is nothing but coloured rectangles on white, one colour per subject, each rectangle moving the way you want that subject to move. The project points you at a companion tool, VACE, which can track boxes off a reference clip or take a pair of boxes directly. Prompt the scene, subject and camera in words, and let the boxes handle position.
Save scribble for last, because it demands the most from you. Rough lines carry the most intent of the three and are the hardest to draw consistently across frames.
Three settings you should not improvise on. Use the config file the 2.0 card names, minimax_h3_control_inpaint_post_norm.yaml, every time. Keep the guidance setting at 1.0; the model was trained expecting that, and raising it applies the effect twice and makes the output worse. And describe scene, subject and camera in the prompt even when the control clip seems to say everything, because the card is explicit that detailed prompts give more stable results.
Where it breaks
The silent failure is the one that will cost you a day. Load version 1's config file against the 2.0 weights and you do not get an error. You get a render. The card spells out what happens: the loader builds half the steering structure, drops the rest as unrecognised, and misplaces what's left. The output looks plausible and is wrong. If you used version 1 and update, changing that one filename is the first thing to do, before anything else.
Boxes carry no posture. The layout input tells the model where a subject is and how big. It says nothing about what the subject is doing. You can get a figure standing in exactly the right spot performing entirely the wrong action, and the fix is in the prompt, not the boxes.
Your control clip decides the shape of the output. The aspect ratio is inherited from your control clip, fitted to a pixel budget you set separately. The length snaps down to a value the decoder accepts, capped at fifteen seconds, at a fixed 24 frames per second. If you need 30 or 60, this is not your tool.
No hosted demo exists, for 2.0 or in practice for version 1 either. There is a showcase Space built around version 1's five sample pairs, but it is paused as I write this and would need its author to restart it. Even running, it carries the older five-input weights, not the eight you came for. There is no free way to try scribble, layout or gray before committing hardware to it.
The licence is not open in the way the word usually means. The add-on ships under the MiniMax H3 Community License Agreement, and the card specifically points at territorial restrictions and an acceptable-use policy. Read it before you build a paid deliverable on top of this. A permissive tag on a small add-on never overrides the terms of the base model it attaches to, and that mistake is getting more common as the add-on layer grows.
What I'd watch next
The interesting question isn't whether these three inputs work. It's whether the shape sticks.
For two years the control story in generative video has been about extraction: run a detector over footage, feed the measurement back in. This release is the clearest signal yet that the other direction works too, where a person draws the constraint by hand and the model treats the drawing as the instruction. If that holds up, the skill that matters for AI video stops being prompt phrasing and starts being blocking, which is a skill a lot of working directors and storyboard artists already have and have been told is irrelevant here.
Try the gray pass first. It's the one that costs you an export and tells you the most.
Medium metadata
Title: You Can Now Direct an AI Video Model With a Drawing Subtitle: MiniMax H3's new control pack adds scribble, box-layout and grayscale input. That sounds like a minor version bump. It changes who the tool is for. Tags: AI Video, Generative AI, Filmmaking, Creative Tools, Open Source Est. read: 8 minutes