We present additional results complementing the main paper. In particular, we show interactive relighting results, using pre-computed images generated by our method and baselines.

Hover your mouse over the images to see animated relighting results!

Table of Contents

1. Additional qualitative results (extends fig. 5)
2. Parameters control (extends sec. 4.6)
3. Light radius
4. Multiple lights
5. Background relighting
6. User study interface (extends sec. 4.5)

1. Additional qualitative results

We present 20 additional randomly-selected results on the "user-controlled" dataset, extending the qualitative results from fig. 5. In this case, we rendered 8 light directions instead of 5 (as used for the user study), in order to also show results where the shadow is behind the object.
Move your mouse from left to right over the images to see the light direction change.

2. Additional results for parameters control

We show how the two main parameters of our method can be adjusted for enhanced artistic control, extending sec. 4.6. Here, both parameters are modified separately, but they could be modified simultaneously for optimal results.

2.1. Guidance scale γ

Here, we show control over the guidance scale γ, extending fig. 6. Increasing the guidance scale generally makes the dominant light source stronger.
Move the mouse vertically to adjust the guidance scale, and horizontally to adjust the light azimuth angle.

The current light azimuth angle is 0°.

The current guidance scale is γ=3.0.

2.2. Latent mask weight β

Here, we show control over the latent mask weight β, extending fig. 7. Increasing the latent mask weight makes the dominant shadow darker.
Move the mouse vertically to adjust the latent mask weight, and horizontally to adjust the light azimuth angle.

The current light azimuth angle is 0°.

The current latent weight is β=0.05.

3. Light source radius

We adjust the light source radius used to generate the coarse shadow fed to SpotLight. Small radii lead to hard shadows, whereas higher radii generate more diffuse shadows.
Move the mouse vertically to adjust the light source radius, and horizontally to adjust the light azimuth angle.

The current light azimuth angle is 0°.

The current light radius is 1 (default).

4. Multiple lights

In our paper, we analyze all the methods by conditioning on a single dominant light source, which is sufficiently realistic in most cases. Here, we show that we can combine outputs from SpotLight at different light directions to simulate multiple light sources. We combine a static light direction (shadow to the right of the object), with a dynamic direction (hover over the images to move this virtual light). We combine the two lightings in linear space (by using the gamma of 2.2) using the following equation: $$x_{\text{combined}}=(0.5 \times {x_{\text{light 1}}}^{2.2} + 0.5 \times {x_{\text{light 2}}}^{2.2})^{\frac{1}{2.2}}.$$ Notice how the static shadow and shading have a clear effect on the combined output.

5. Background relighting

In our work, we use SpotLight for object relighting. We experiment with extending SpotLight for full-scene relighting using the following approach.

1. We run SpotLight as-is to render the object lit with a desired shadow. We obtain the SpotLight output.
2. We run the backbone (in this case ZeroComp), with the inverted shading mask, feeding it only the shading of the object and its shadow (obtained by dividing the SpotLight prediction with the albedo map) and letting the backbone shade the background. We obtain the relit background output.
3. We repeat step 2 for each of the 8 light directions, and average them, giving us the average relit background.
4. We find that the background relighting in step 2 and 3 poorly reconstruct the background's identity. We therefore obtain a per-pixel highlight map, by dividing the relit background prediction with the average relit background. We multiply that highlight map with the original background and obtain a more accurate output, relit background + composition.

We observe in the following example that background shadows and highlights are added in a consistent manner with respect to the object's shadow.

Note about the checkpoint used for background relighting

In step 1, we use the same ZeroComp checkpoint as in all other experiments. For steps 2 and 3, in order to properly relight the background, we found that this checkpoint had limited full-scene relighting capabilities. We hypothesize that this is due to the fact that the neural renderer backbone is trained to relight only a small region of an image (circular and rectangular masks at training time). We therefore train a separate model from scratch for 270K iterations on inverted shading masks, where we take the inverse of the circular and rectangular masks for shading masking. Furthermore, we only keep the largest connected component in the mask. This leaves training examples where only a small region of the shading is known and the full background lighting needs to be inferred.

6. User study interface

Here, we show the user interface used to perform our human perceptual study, as described in sec. 4.5 in the paper. The user is first shown an instructions page, followed by the questions page (which is the same for the 120 2AFC questions).