Analysis September 10, 2026 workflow 10 min read

ID-V2V treats identity preservation as a relighting problem, which is why a mismatched stylized keyframe still works and why the workflow is reproducible from one video.

ID-V2V Eyeline Labs SAM3 Wan 2.1 video-gen ai-editing creative-workflows open-weights

ID-V2V Lets Your Keyframe Be Wrong

A Netflix and Eyeline Labs model restyles footage you already shot from one hand-painted frame. The interesting part is that the frame does not have to match.

Every keyframe-driven video tool has the same unwritten requirement: the keyframe has to line up. You paint frame zero in an image editor, and if the pose drifted or the expression changed while the editor was restyling it, the whole clip inherits that drift.

ID-V2V's documentation says the opposite. Feed it a stylized first frame that does not match your source, and frame zero follows your painted version while every frame after it re-aligns to the source video's identity and performance. The mismatch corrects itself.

That sounds like a bug report written as a feature. It is neither. It falls directly out of how the model was built, and once you see the reason, the rest of the workflow stops looking like a pile of shell scripts and starts looking like a design.

Before going further, a correction to something you may have seen. ID-V2V surfaced in a lot of feeds this morning because a personal account reuploaded the weights to a fresh Hugging Face repo. That upload is hours old. The work is not. The official Eyeline-Labs/ID-V2V repo has a creation date of July 15, the GitHub repo was created July 14 and last pushed July 29, and the paper went to arXiv on July 24. This is eight-week-old research that most people missed, which is a better reason to write about it than a launch would be.

The trick: identity preservation is a relighting problem

Here is the obstacle the team names in the paper. To train a model that restyles a video while keeping a performer's face intact, you need pairs: the same performance, same expressions, same eye line, rendered in two different styles. Those pairs barely exist. You cannot scrape them.

So they reframed the problem. Their stated insight is that facial appearance and expression should stay invariant when you restyle a shot, and that illumination is the primary permissible variation. A face in a neon-lit alley and the same face in an overcast field is the same face, differently lit.

Cast that way, identity preservation becomes video relighting, and relighting pairs you can make from a single video. That one move eliminates the paired-data problem entirely. Training data comes from one clip.

It also explains the keyframe behavior. The model is not copying your painted frame across the timeline. It is holding the source performance as the thing that must survive and treating your keyframe as a specification of scene, lighting and style. Your keyframe describes the world. The source video describes the person. When they disagree about the person, the source wins from frame two onward, because that is what the model was trained to protect.

Analogy that actually maps: it is closer to a colorist's grade than a filter. A grade changes how the light behaves and leaves the performance alone. That is the operation here, extended from color into full scene regeneration.

What actually ships versus what the paper describes

Worth flagging, because it will confuse you if you read the paper first.

The abstract describes a framework combining relit facial regions, facial normal maps, edited keyframes and depth sequences. The repo counts things differently, and the difference is worth understanding before you pick a checkpoint.

Keyframes are not one of the swappable signals. They are the always-on input that drives every recipe and both checkpoints identically. What varies is the VACE control conditions, and the repo's own table counts those as 1 for the default idv2v (foreground-on-gray pixels, meaning the person segmented by SAM3 with the background grayed to nothing) and 3 for the alternate idv2v_with_normal_depth (that same signal plus surface normals from DAViD and depth from DepthAnything-V2).

So the recommended default is the simpler one, and the repo is specific about when to reach for the other: only when the source video's depth already matches the scene you intend to generate. Otherwise you are constraining regions you wanted the model to invent.

One thing I want to be careful about, because it would be easy to state too confidently. The abstract's "relit facial regions" and the repo's "foreground-on-gray pixels" are not obviously the same object. The README describes relighting as something the model does to the masked region during generation, saying the default checkpoint "keeps (relights) whatever is inside the SAM3 mask." That reads as an output property, not a pre-relit input. Whether the shipped single-condition path is the paper's relit-face signal under a plainer name, or a simplification of it, is not something either document settles.

The two checkpoints share an architecture. Load the wrong one into the wrong script and nothing errors. You just get worse video. The maintainers flag this themselves in two separate places, which tells you how often it happens.

Put this into practice

Step 0: try it in a browser before you download 96 GB. There is a hosted Space, hugging-apps/id-v2v, running on ZeroGPU A10G hardware with three bundled examples. That is a far smaller card than the 8× A100-80GB the repo was tested on, so treat it as a look at whether identity preservation holds on your kind of footage, not as a performance test. It takes five minutes and it will tell you whether the next hour is worth spending.

Step 1: set up. One uv environment covers preprocessing and inference.

git clone https://github.com/Eyeline-Labs/ID-V2V && cd ID-V2V
uv sync && source .venv/bin/activate

Step 2: authenticate, then download. SAM3 is gated on Hugging Face, so this fails confusingly if you skip the login. Do it first.

hf auth login                          # token needs read access to facebook/sam3
bash scripts/download_checkpoints.sh   # idv2v.pth + SAM3 + Wan2.1 T5, VAE, tokenizer, CLIP

That pulls roughly 96 GB. Start it and go do something else. Adding --with-depth fetches the alternate checkpoint plus DAViD and DepthV2, about 75 GB more, and you do not need it yet.

Step 3: build a sample directory. This is the whole input format, and it is refreshingly small.

my_sample/
├── source.mp4                 your footage, any length
├── stylized_first_frame.png   frame 0, restyled in any image editor
├── prompt.txt                 what the new scene is
└── keyframes/                 optional
    ├── 40.png                 pins frame 40
    └── 80.png                 pins frame 80

Make stylized_first_frame.png however you like. Pull frame zero, take it into whatever image editor you use, change the world around the person. The repo names NanoBanana as an example. Do not fuss over matching the pose, for the reason in the section above.

Step 4: generate. There are two documented paths and you want one of them, not both. The recipe script is the one-liner, and it already runs SAM3 preprocessing internally before generating:

SAMPLE_DIR=test_samples/restylization/two_sitting_woman bash scripts/examples/restylization.sh

The generic path is the two-step version, and it is what you need if you want to change what gets segmented:

SAMPLE_DIR=my_sample SAM_PROMPT=head bash scripts/preprocess.sh
SAMPLE_DIR=my_sample bash scripts/infer.sh

SAM_PROMPT is the text prompt SAM3 uses to decide what to segment. It defaults to person, and the repo names head and dog as alternatives. That string determines which pixels become the control condition, so it decides what the model is anchored to and what it regenerates freely, which makes it the most consequential setting in the workflow and the least obvious one. The trap: scripts/examples/restylization.sh uses its own default prompt, so setting SAM_PROMPT on that line does nothing. If you want anything other than person, take the two-step path.

For multiple GPUs, set GPU="0,1,2,3,4,5,6,7" and it launches torchrun with sequence parallelism. For one card, GPU="0" runs plain Python with CPU offload, which the repo describes only as slower.

Step 5: look at flip_test.mp4 before anything else. The run writes several outputs, and this is the one to open first. It plays your generated video and freezes at up to five evenly spaced frames, alternating between the generated frame and the source frame with a colored label on each. A built-in A/B, shipped by default. Most research repos make you build this yourself, and having it there changes how fast you can judge a run.

Step 6: try relighting, which needs no preprocessing at all. This is the cheapest recipe and it is a different tool than it first appears.

SAMPLE_DIR=test_samples/relighting/two_sitting_woman bash scripts/examples/relighting.sh

Relight only frame zero in your image editor, keeping the background and everything else exactly as shot, then feed it in raw. No SAM3, no mask, no grayed background. The unmodified source video becomes the control signal, so the whole scene survives and only illumination changes. Given that the model's core idea is relighting, this recipe is the model doing the thing it was actually trained on, with nothing layered over it.

Longer than 81 frames? Nothing special to configure. MAX_NUM_FRAMES sets the length, generation happens in overlapping 81-frame clips, and each clip conditions on the end of the one before. The repo's advice is worth taking: use more keyframes as the source gets longer.

Honest limitations

The hardware bar is the real gate. Tested on 8× A100-80GB. There is a single-GPU path and the repo will not tell you how slow it is, which usually means slow enough that they would rather you not ask. Add ~96 GB of checkpoints, plus free RAM at least the size of the checkpoint if you leave STAGE_CHECKPOINT_TO_SHM at its default true. This is not a laptop workflow and nobody claims it is.

"Research exploration, for demonstration and inspiration purposes only." That is the authors' phrasing, on both the model card and the project page. The weights are Apache 2.0, so the license itself is permissive, but the stated intent is not production, and there is a gap between those two things that your client will notice before you do. Worth a conversation before you build a deliverable on it.

It is human-centric by construction. The training insight, that appearance and expression stay fixed while illumination varies, is a claim about faces. SAM3 defaults to segmenting people. If your restylization job is a product shot or an empty exterior, this is the wrong tool and no flag makes it the right one.

Multi-clip drift is managed, not solved. Each clip conditions on the previous clip's end frame, described as drift-controlled, not drift-free. On a long source the repo's own recommendation is to pin more keyframes, which is another way of saying the further you get from an anchor the more it wanders.

The silent checkpoint mismatch is a bad failure shape. Two files, same architecture, no error, worse output. If your results are mediocre and you cannot work out why, check which .pth you loaded before you change anything else.

Every number in the paper is the authors' own. Extensive experiments showing it significantly outperforms existing methods are the team's experiments. No independent reproduction that I could find.

What I would test first

The claim I most want someone to push on is the imperfect-keyframe one, because it is the claim that would change how you work.

If it holds strongly, the image-editing step stops needing care. You stop trying to make your restyled frame match the source pose, you stop rerolling the editor until the expression is right, and you just paint the world you want. That saves the slowest part of the loop.

If it holds weakly, you have a tool with a hidden tolerance and no documented edge, and you will find that edge in the middle of a job.

So the test is to break it on purpose. Take a source clip, restyle frame zero, and deliberately push the pose further and further off: head turned a bit more, then a lot more, then looking the other way entirely. Somewhere in that sweep the re-alignment stops working, and knowing where that point sits on your kind of footage is worth more than any benchmark in the paper.

If you run that sweep, I want to hear where it broke. Nobody has published that number, and it is the one that decides whether this is a research demo or a tool.

Medium metadata

Suggested kicker: They solved the missing-training-data problem by deciding that identity preservation is really a lighting problem.

Primary sources:

Official model card and checkpoint notes: https://huggingface.co/Eyeline-Labs/ID-V2V
Code, recipes and input format: https://github.com/Eyeline-Labs/ID-V2V
Paper (arXiv 2607.22830, submitted 24 July 2026, SIGGRAPH Asia 2026): https://arxiv.org/abs/2607.22830
Project page: https://eyeline-labs.github.io/ID-V2V/
Hosted Space: https://huggingface.co/spaces/hugging-apps/id-v2v