A 28 MB File Separates the Light From the Paint in Any Photograph
ShadeNet-2 takes one photo and hands back four images: the colours with the lighting removed, a depth map, a surface map, and the light on its own. It runs in a browser tab for free, and the lighting layer was never taught what light looks like.
There is a model on Hugging Face that is 28 MB and was trained in seventeen hours on a graphics card from 2019. You give it a picture. It gives you back the same picture split into parts: what colour each surface actually is, how far away it is, which way it faces, and how much light is landing on it.
That last one is the interesting bit, because nobody ever showed it a correct answer.
The lighting layer has no training data. None. Its only instruction during training was that the colours multiplied by the light had to reconstruct the original photograph. That one rule, applied eight thousand times, was enough to teach it the difference between a red wall and a white wall in red light.
I want to talk about why that constraint works, because once you understand it you will know exactly where the model falls apart, and you will know it before it ruins a day of work.
What you get and why you would want it
ShadeNet-2 went up on 15 September under an Apache 2.0 licence, which is the permissive kind with no commercial restriction. One photo goes in. Eight channels come out, which unpack into four usable images:
Base colour. The paint, with the lighting factored away. A white mug photographed under a warm lamp comes back white instead of cream.
Depth. How near or far things are, relative to each other.
Surface direction. Which way each bit of the surface is pointing. This is what lets a renderer work out how a new light would hit it.
Light. A grey image of the illumination on its own.
If you have ever hand-painted a base colour layer to take the shine off a photographed product so you could relight it in a comp, this is that job, done in a few seconds, from one file.
For texture artists, it means a photograph of a brick wall becomes usable material maps instead of a picture of bricks with their lighting baked in. For compositors, it means you can pull the original light out, put a new one in, and have the surface directions to make the new light behave. For anyone doing matte work, the depth and surface maps arrive in the same pass and lined up with the colour, so you are not chasing three tools into registration.
Why the no-data trick works
Say you photograph a grey ball under an orange light. The pixels are orange. There are two explanations: an orange ball under white light, or a grey ball under orange light. From one photograph, both are true.
A model trained only on labelled examples learns to guess which is more common. This one is trained on a constraint instead. It is required to output a colour layer and a light layer whose product is the photograph. That forces a split. It has to put something in each bucket, and the only way to get the multiplication right across thousands of images is to start behaving as though objects have consistent colour and light varies.
The author calls this reconstruction coupling and says outright that it is the shading head's only supervision.
And this is exactly where it breaks, which is the good part. The light layer is a single grey channel. Grey has no hue. So if the actual light in your scene is coloured, the model has nowhere to put that colour except into the base colour layer.
The card names this: the shading assumes white light, and strongly coloured illumination leaks into the base colour. Sunsets and neon are listed by name.
You do not have to discover that failure by wasting an afternoon on a golden-hour plate. It follows from the shape of the thing. One grey channel cannot hold a colour, so the colour goes somewhere else.
Put this into practice
The lowest-friction version takes about ninety seconds and needs nothing installed.
1. Open the demo. The author is running a free hosted version that went live on 15 September. Upload a photo, get the split back. Try three of your own plates before you download anything. If your work is mostly warm interiors, you will find out here rather than later.
2. Run it locally if the demo convinces you. Two small packages and one file:
pip install onnxruntime pillow numpy
Then grab onnx/model_quantized.onnx from the repo. It is 28 MB and it runs on your processor. No graphics card involved.
3. Feed it correctly. The input is an array shaped [1, 3, height, width] with values scaled between minus one and one. Sizes that are multiples of 32 work best. Out comes [1, 8, height, width]:
channels 0-2 base colour
channel 3 depth (relative)
channels 4-6 surface direction
channel 7 light
There is also a command-line path if you would rather not write anything, but it is the separate PyTorch route rather than the small one above. Different install, different file:
pip install torch torchvision pillow numpy
python inference.py photo.jpg --output_dir ./output
That one reads checkpoints/shadenet2.ckpt (333 MB) instead of the 28 MB file. Pick one path and stay on it.
4. Check the multiplication. Take the base colour and the light and multiply them together. You should get something close to your original photo. Where it does not match, the model has misread the scene, and that mismatch is a free quality check. Run it on every plate. It costs one line and tells you whether to trust the split.
5. Take the maps into your comp. The base colour is your relightable layer. The surface directions drive how a new light falls. The depth gives you atmospheric separation. And if you want to keep a bit of the original lighting, blend the light layer back in at partial strength rather than throwing it away.
That is the whole workflow. Roughly ten minutes end to end if you already have somewhere to put the maps.
Where it breaks
I have been specific about the colour-light failure because it is the one that will cost you. Here is the rest, all of it published by the author rather than found the hard way.
Depth is relative, not measured. It tells you what is nearer. It does not tell you how much nearer. Do not read distances off it and do not try to build geometry from it.
Surface directions go soft in foliage and sky. The author says these are the hardest of the four maps, and the reason is honest: the training labels for those regions were noisy too. Anything with a lot of leaves or an open sky will give you mush where you wanted detail.
There is no confidence output. When the model is wrong, it is wrong with exactly as much conviction as when it is right. The author's phrasing is that confidently wrong training labels get fitted confidently. This is why the multiplication check in step four earns its place.
The training pictures were snapshots. It learned from 8,077 Flickr photographs, labelled by another model rather than measured. Ordinary scenes, ordinary cameras. A studio product shot on a white sweep is a different kind of image from anything it saw, and a rendered frame is further still. It may work on both. It was not trained for either.
It is small, and small shows. Twenty million parameters is tiny. Fine detail is not this model's job. Treat the output as a good starting layer, not a finished pass.
None of that makes it less useful. It makes it plannable, which is the thing that separates a tool you can put in a paid job from one you can only play with. I would rather have a model that tells me it cannot handle my neon sign than one that hands me a confident wrong answer with no warning attached.
The part worth arguing about
The author published the full validation table, the failure modes in plain sentences, and the fact that the whole thing cost seventeen hours on a consumer card. That combination is more interesting to me than the model.
Because the skill on display is not scale. It is knowing which constraint to impose. Somebody worked out that "colour times light equals the photo" is a strong enough rule to teach a network something it has no examples of, and then built the smallest thing that could hold that rule. The result runs on a laptop processor for free.
Which makes me wonder what else in a creative pipeline is sitting there waiting for the right constraint instead of the right dataset. Focus separation. Reflection and transmission on glass. Motion and texture in a blurred frame. Every one of those is a split where you already know what the parts have to add back up to.
If you try this on footage rather than stills, frame by frame, I want to hear whether the split stays stable across a shot or flickers. That is the question the model card does not answer, and it is the one that decides whether this is a still-image toy or something that belongs in a moving-image workflow.
Medium metadata
Title: A 28 MB File Separates the Light From the Paint in Any Photograph
Subtitle: ShadeNet-2 takes one photo and hands back four images: the colours with the lighting removed, a depth map, a surface map, and the light on its own. It runs in a browser tab for free, and the lighting layer was never taught what light looks like.
Tags: VFX, Photography, Generative AI, Compositing, Open Source
Suggested kicker: Its lighting layer had no training data. One constraint was enough, and that constraint is also the reason sunsets ruin it.