RenderFormer Renders Global Illumination With No Ray Tracing. The Weights You Can Download Are Not the Model in the Paper.
A transformer that replaces the path tracer, ships under MIT, and comes with a set of caveats nobody is repeating.
Hand RenderFormer a triangle mesh with materials and lights and it gives you back a globally illuminated image. Soft shadows, specular reflections, indirect bounce, color bleeding off a red wall onto a white sphere. There is no ray tracing anywhere inside it. No sample count to raise, no fireflies to clean up, no denoiser, and no per-scene optimization step. You do not train it on your scene. You hand it geometry it has never seen and it returns a finished frame in a fixed, short amount of time.
That is a strange thing for a renderer to be, and the strangeness is the point. Most AI in a 3D artist's pipeline sits next to a real path tracer and helps it: OptiX and Open Image Denoise clean up a noisy render that Cycles actually computed. RenderFormer computes the transport itself, in one forward pass, with attention instead of rays.
Then there is the part that sent me back through the primary sources twice. The speed figures circulating this week are not from the model you can download. They belong to RenderFormer-V2, whose paper states that its trained model and code "can be found at" its project page, where the Model button goes to a bare Hugging Face organization listing containing two checkpoints, both of them V1.
Why a renderer without rays is worth your attention
Path tracing has one shape of cost. You trace rays, you gather samples, and the noise falls off with the square root of how many you gather. That is why quality and time are the same dial, and why every optimization in the field for thirty years has been about getting more useful information per ray.
A learned renderer breaks that relationship. Because transport is predicted rather than sampled, there is no variance to converge, which means no sample count, no noise, and no denoiser artifacts around thin geometry. Render time stops depending on how clean you want the image and starts depending only on how much geometry and how many pixels you handed it.
For a working artist that is a different tool, not a faster one. A path tracer gets better if you wait. This does not. It gives you the same frame in the same time every run, which makes it a preview instrument, not a final-frame renderer. The interesting question is whether a preview instrument that models real indirect light beats a rasterizer that approximates it, and on that question RenderFormer is the only entrant you can actually download and try.
The mechanism: triangles as tokens, rays as tokens
Two transformer stages, and the split between them is the whole design.
The first stage is view independent. Each triangle in the scene becomes one token. Its three vertex positions get concatenated into a nine-dimensional vector and encoded with an adapted rotary embedding using six frequencies spread from 1.0 to 5.0. Its material, a GGX microfacet BRDF with diffuse albedo, specular albedo, roughness and emission, becomes a ten-dimensional vector projected up into the token. Normals get a NeRF-style frequency encoding. Then full bidirectional self-attention runs over the whole set, plus a handful of register tokens.
That attention is the light transport. Every triangle sees every other triangle and the network works out how light moves between them. There is no camera in this stage at all, which is why the result can be reused across views of the same scene.
The second stage is view dependent and it is where the picture appears. The camera's rays get bundled into groups of 64, one bundle per 8 by 8 pixel patch, and each bundle becomes a token. Those ray tokens cross-attend into the triangle tokens from stage one, then a dense prediction head over the last few layers turns them into HDR pixel values, stored as log(x+1) so a linear network can cover the range of real light.
The analogy that made it click for me: stage one is a lighting bake, and stage two is a lookup into that bake from wherever you put the camera. The difference from an actual bake is that no lightmap UVs exist, nothing is precomputed per scene, and the "bake" is a set of attention activations that the paper measures at 18.6 to 42.9 milliseconds across its four test scenes.
The released large checkpoint is not the paper's architecture
This is the finding I did not expect, and it changes how you read every benchmark you see quoted.
The SIGGRAPH 2025 paper describes a network with 768-dimensional tokens, 6 attention heads, 16 register tokens, and a 12-plus-6 layer split using full self-attention throughout. Table 1 lists that configuration at 205M parameters.
The flagship weights on Hugging Face are microsoft/renderformer-v1.1-swin-large, 483,269,903 parameters. Its config.json reports latent_dim: 1024, num_heads: 8, num_register_tokens: 32, and a 12-plus-12 layer split with view_transformer_use_swin_attn: true. Windowed attention in the view-dependent stage, not the full attention the paper analyzes.
So the large model is a different architecture wearing the paper's name. renderformer-v1-base is the paper's configuration exactly, 205,173,391 parameters at 768 dimensions, 6 heads, 16 registers, 12 plus 6 layers, no windowed attention. It went up five minutes before the large one, both on 2025-05-15. The repository has not been pushed since 2025-09-25 and has no 2026 commits, which is worth stating plainly because a lot of coverage this week reads as though this landed in August 2026. It did not.
Watch the default when you run inference. Despite a commit titled "set default configs to base-200m," infer.py still defaults --model_id to the swin-large checkpoint, so the model you get without thinking about it is the one the paper does not describe.
The repo is also honest about a second gap, tucked inside a collapsed block most readers will never open:
"We found a shader bug in the training data that we used in the submission. We re-trained the models with the corrected shader and released the new models. Thus the model performance and output might be different from the results in the paper."
That applies to both checkpoints. So base matches the paper's architecture but not its training data, and large matches neither. Every quality number in the SIGGRAPH paper describes something you cannot download. The paper is a good description of the idea. It is not a spec sheet for the artifact.
What the speed numbers actually say
The paper's Table 2 measures at 512 by 512 on a single NVIDIA A100, with what the authors call an unoptimized pure PyTorch implementation, timings taken with pre-cached kernels and scene loading excluded. Four scenes, between 4,400 and 7,321 triangles:
| Scene triangles | Cycles, 4096 adaptive spp + denoise | RenderFormer |
|---|---|---|
| 5,366 | 3.97 s | 0.0760 s |
| 4,400 | 4.73 s | 0.0613 s |
| 4,527 | 3.77 s | 0.0625 s |
| 7,321 | 2.71 s | 0.0978 s |
That works out to roughly 28x to 77x against denoised adaptive Cycles, and 80x to 183x against raw 4,096 samples per pixel. Real, and measured on the same hardware for both sides, which is more than most comparisons manage. The paper also runs an equal-time comparison against Cycles at 26 samples per pixel with no denoising and no adaptive sampling, which is the fairer preview-versus-preview test, and RenderFormer wins that one on noise by construction.
The caveat that matters more is hardware. These are A100 numbers. I went looking for a single published timing on Apple Silicon and there is none, in the README, on the project page, or in either paper. The repo says the code is tested on Apple Metal GPUs and that "The minimal GPU memory requirement is 8GB," but it never puts those two statements together, and infer.py overrides your precision flag on MPS with the warning that "bf16 and fp16 will cause too large error in MPS, force using fp32 instead." Full precision weights on a machine whose memory floor was probably measured at half precision. Anyone telling you what this does on an M-series Mac is guessing, including about whether 8GB is enough there.
The version everyone is quoting has a paper but no weights
RenderFormer-V2 has a project page and an 18-page paper you can download from it, formatted for ECCV. The claims are large. 128,000 primitives at evaluation against V1's ceiling near 4,000, with training running to a 64,000-primitive budget. 2048 by 2048 output with no upsampling stage. Refraction, volumetric scattering, HDR environment lighting, textured and displaced surfaces, and a learned 9-dimensional material space replacing the hard-coded GGX BRDF. Roughly 10 million procedurally generated scenes, about 70 TB, 19 days on 32 A100s.
Three things the coverage is dropping.
It is not a Microsoft paper. Three of five authors are at Stanford, with Microsoft Research and William & Mary as collaborators. V1's Hongzhi Wu and Xin Tong are gone from the author line, and Lvmin Zhang and Maneesh Agrawala are added. The ECCV 2026 venue appears throughout the page but there is no arXiv listing to corroborate it, and the page's own arXiv button links to https://arxiv.org/abs/ with nothing after it.
The speed advantage depends heavily on where you stand. On a 64K-triangle scene at 512 by 512 the page reports 0.525 seconds for V2 against 5.13 for Cycles. Push to 4096 by 4096 and it is 14.9 against 216.8. By primitive count at fixed resolution, 1,597 primitives gives 0.047 against 3.63, but at 127,233 primitives the gap narrows to 1.12 against 5.25. That is a spread from roughly 5x to roughly 77x, and a single headline multiple hides which end you are on.
And the weights do not exist. Not under the Microsoft organization, not under any other organization on Hugging Face. The paper says otherwise. That is the sharpest thing in this whole story: a paper asserting a release, pointing at a page whose Model button opens an organization listing where the thing is not.
There is also a third paper in this lineage nobody is mentioning, RenderFormer++ from a USTC group, which scales the same idea independently and has likewise released no code. The pattern is becoming familiar.
Put this into practice
Assume an hour, most of it environment setup. You need a machine with at least 8GB of GPU memory, and the code is tested on Linux, macOS and Windows.
1. Set up the environment. Install PyTorch 2.0 or newer yourself first, because it is not in requirements.txt. Then:
git clone https://github.com/microsoft/renderformer
cd renderformer
pip install -r requirements.txt
python3 -c "import imageio; imageio.plugins.freeimage.download()"
That last line pulls the FreeImage plugin, which you need for HDR output. No Python version is stated anywhere in the repo, which is the first small annoyance. Start on 3.10 or 3.11.
2. Render the Cornell box. Convert a scene description into the HDF5 format the model reads, then infer:
python3 scene_processor/convert_scene.py examples/cbox.json --output_h5_path tmp/cbox/cbox.h5
python3 infer.py --h5_file tmp/cbox/cbox.h5 --output_dir output/cbox/
You get cbox_view_0.exr, the linear HDR result, and cbox_view_0.png, an LDR version. The default tone mapper is none, so that PNG will look flat and wrong until you pass --tone_mapper agx, filmic, or pbr_neutral. Add --model_id microsoft/renderformer-v1-base if you want the architecture the paper actually analyzes.
3. On a Mac, force the attention fallback. Flash Attention is CUDA only. The script falls back to torch scaled dot product attention on its own, but making it explicit skips a confusing warning:
ATTN_IMPL=sdpa python3 infer.py --h5_file tmp/cbox/cbox.h5 --output_dir output/cbox/
Expect fp32 regardless of what you ask for, per the MPS override above.
4. Bring your own geometry through Blender. Install iamNCJ/renderformer-blender-extension. It adds a sidebar panel with Initialize All to drop in a template scene with walls, floor, light triangles and a camera, Normalize Scene, a scene stats panel showing your triangle count, and Export Scene to HDF5 and Export Animation to HDF5. It does not render. You still run infer.py yourself. Fair warning that the extension's own install link reads "Download the latest release from here", so you are cloning and installing the folder by hand.
5. Remesh anything dense before you try. The included script handles it:
python3 scene_processor/remesh.py --input high_res_mesh.obj --output remeshed.obj --target_face_num 1024
The README prints that flag as ----output with four dashes. That is a typo in the docs. The real argparse flag is --output.
6. Stay inside the training envelope. This step decides whether your render looks like the gallery or like garbage. From the repo, verbatim: camera distance to scene center in [1.5, 2.0], field of view in [30, 60] degrees, scene bounding box in [-0.5, 0.5] on every axis, up to 8 light triangles each scaled [2.0, 2.5] at distance [2.1, 2.7] with emissions summing to between 2500 and 5000, and total triangles up to 4,096 with 8,192 usually still working.
7. Get a ground-truth comparison for free. The repo ships a Blender exporter so you can render the same scene properly and see what the network got wrong:
python3 scene_processor/to_blend.py examples/cbox.json --output_dir tmp/cbox --dump_blend --save_img --spp 4096
Do this on your second scene. Seeing the two side by side teaches you the failure modes faster than reading about them.
Honest limitations
The operating envelope is not a footnote. It is the product.
Eight light sources, maximum, white only. The paper says colored light has its color ignored outright, and that more than eight lights produces "incomplete shadows or missing highlights." Lights inside the scene fail. Camera inside the scene fails. Geometry behind the camera fails. Two of those have workarounds the authors endorse and I have not seen anyone mention: render each light separately and composite, and render three per-channel passes and blend for colored light. Both cost you a forward pass each, which at 60 milliseconds is affordable.
Specular is white only, and diffuse plus specular must sum to no more than 1.0. Emission is white only, and only on light-source triangles.
No texture authoring path. The scene JSON assigns material parameters per triangle, so you cannot bring a textured asset in through the documented route. The model itself does take a 13-channel per-triangle texture tensor, so the plumbing is there, but the paper describes textured results as blurred and the tooling does not expose it.
Around 4,096 triangles, extending to 8,192 in practice. That is a single simplified prop, not a scene. Objects need to be watertight and QSlim-simplified with roughly uniform triangle sizes.
512 by 512 by default, because the camera was placed outside the bounding box at fixed 512 resolution during training. Higher resolutions do work, and the paper says the model "fails gracefully, with most of the errors focused around depth discontinuities," which is a polite way of saying edges go soft.
Specular is correct to roughly three bounces. Higher-order bounces are dropped.
Temporal coherence is not enforced. The model trains on single frames, and the authors say it holds up across parameter changes in their supplemental video, but nothing in the architecture guarantees frame-to-frame stability.
Who should skip this entirely: anyone with a textured production asset, anyone who needs colored or numerous lights and does not want to composite passes by hand, anyone rendering interiors with the camera inside the room, and anyone whose meshes are dense enough that remeshing to 4,096 faces would destroy them. That is most commercial 3D work.
Who should spend an hour on it: anyone who wants to see what a scene-generalizing neural renderer feels like to drive, on hardware they already own, with MIT weights and no licensing trap waiting at the end.
What I would watch instead of waiting
RenderFormer V1 is a research artifact you can play with, not a renderer you can work in. V2 is a paper whose weights do not exist. Neither of those is a reason to look away, because the V1 checkpoint, tight envelope and all, is the only demonstration with public weights that a network can learn light transport across scenes it has never seen and produce something that reads as correct. Everything else you can actually install is a denoiser on top of a real path tracer, a rasterized approximation, or a game-engine SDK. In the papers it is a different story: NVIDIA and UCSD published a generalizable light transport embedding at SIGGRAPH 2026 that predicts indirect light for unseen indoor scenes at far larger scale, and it too has no public weights.
That is the pattern worth naming. Four separate groups have now shown that learned transport generalizes. Exactly one of them has shipped weights, fifteen months ago, and has not pushed code since September. The bottleneck in neural rendering right now is not the research. It is that nobody is releasing.
So the concrete thing to watch is not a benchmark. It is whether the Model button on that V2 page ever starts pointing at a model. If it does, the operating envelope is the number to check first, and I would look at lights before anything else. V2 finally made them colored. It kept the ceiling at eight.
If you run V1 on your own geometry, I want to know where it broke first. My bet is the camera envelope, not the triangle count.
Medium metadata
- Title: RenderFormer Renders Global Illumination With No Ray Tracing. The Weights You Can Download Are Not the Model in the Paper.
- Subtitle: A transformer that replaces the path tracer, ships under MIT, and comes with a set of caveats nobody is repeating.
- Tags: Neural Rendering, 3D Graphics, Machine Learning, Blender, Open Source
- Suggested publication: Tkay Nation's Writes
Sources
- microsoft/renderformer: README, license, prerequisites, scene setting tips, commands
- microsoft/renderformer-v1.1-swin-large: weights,
config.json, parameter count - microsoft/renderformer-v1-base: the paper-configuration checkpoint
- RenderFormer, SIGGRAPH 2025 (arXiv 2505.21925): architecture, training data, Table 2 timings, limitations
- RenderFormer-V2 project page and paper: V2 claims, timings, author list, limitations
- iamNCJ/renderformer-blender-extension: Blender exporter and its material and lighting constraints
- A Generalizable Light Transport 3D Embedding for Global Illumination (arXiv 2510.18189): the SIGGRAPH 2026 alternative with no public weights