Introduction
This work proposes an finish to finish 2D relighting diffusion mannequin. This mannequin learns bodily priors from artificial dataset that includes bodily based mostly supplies and HDR surroundings maps. It may be additional used to relight a number of views and be used to create a 3D illustration of the scene.
Methodology
Given a picture and a goal HDR surroundings map, the purpose is to be taught a mannequin that may synthesize a relit model of the picture which here’s a single object. That is achieved by adopting a pre-trained Zero-1-to-3 mannequin. Zero-1-to-3 is a diffusion mannequin that’s conditioned on view route to render novel views of an enter picture. They discard its novel view synthesis parts. To include lighting circumstances, they concatenate enter picture and surroundings map encodings with the denoising latent.
The enter HDR surroundings map E is cut up into two parts: E_l, a tone-mapped LDR illustration capturing lighting particulars in low-intensity areas, and E_h, a log-normalized map preserving info throughout the total spectrum. Collectively, these present the community with a balanced illustration of the vitality spectrum, guaranteeing correct relighting with out the generated output showing washed out on account of excessive brightness.
Moreover the CLIP embedding of the enter picture can also be handed as enter. Thus the enter to the mannequin is the Enter Picture, LDR Picture, Normalized HDR Picture and CLIP embedding of Picture all conditioning the denoising community. This community is then used as prior for additional 3D object relighting.
Implementation
The mannequin is skilled on a customized Relit Objaverse Dataset that consists of 90K objects. For every object there are 204 photographs which can be rendered beneath completely different lighting circumstances and viewpoints. In whole, the dataset consists of 18.4 M photographs at decision 512×512.
The mannequin is finetuned from Zero-1-to-3’s checkpoint and solely the denoising community is finetined. The enter surroundings map is downsampled to 256×256 decision. The mannequin is skilled on 8 A6000 GPUs for five days. Additional downstream duties reminiscent of text-based relighting and object insertion might be achieved.
Outcomes
They present comparisons with completely different backgrounds and comparisons with different works reminiscent of DilightNet and IC-Light.
This determine compares the relighting outcomes of their technique with IC-Mild, one other ControlNet based mostly technique. Their technique can produce constant lighting and coloration with the rotating surroundings map.
This determine compares the relighting outcomes of their technique with DiLightnet, one other ControlNet based mostly technique. Their technique can produce specular highlights and correct colours.
Limitations
A significant limitation is that it solely produces low picture decision (256×256). Moreover it solely works on objects and performs poorly for portrait relighting.