Laval University logo Adobe logo

Shadow Harmonization for Realistic Compositing

1Université Laval
SIGGRAPH Asia 2023 Conference Proceedings

Compositing virtual objects into real background images requires one to carefully match the scene's camera parameters, surface geometry, textures, and lighting to obtain plausible renderings. Recent learning approaches have shown many scene properties can be estimated from images, resulting in robust automatic single-image compositing systems, but many challenges remain. In particular, interactions between real and synthetic shadows are not handled gracefully by existing methods, which typically assume a shadow-free background. As a result, they tend to generate double shadows when the synthetic object's cast shadow overlaps a background shadow, and ignore shadows from the background that should be cast onto the synthetic object. In this paper, we present a compositing method for outdoor scenes that addresses these issues and produces realistic cast shadows. This requires identifying existing shadows, including soft shadow boundaries, then reasoning about the ambiguity of unknown ground albedo and scene lighting to match the color and intensity of shaded areas. Using supervision from shadow removal and detection datasets, we propose a generative adversarial pipeline and improved composition equations that simultaneously handle both shadow interaction scenarios. We evaluate our method on challenging, real outdoor images from multiple distributions and datasets. Quantitative and qualitative comparisons show our approach produces more realistic results than existing alternatives.


Paper preview

This paper can be cited as follows:
                 title     = {Shadow Harmonization for Realistic Compositing},
                 author    = {Valen{\c{c}}a, Lucas and Zhang, Jinsong and Gharbi, Micha{\"e}l and Hold-Geoffroy, Yannick and Lalonde, Jean-Fran{\c{c}}ois},
                 booktitle = {ACM SIGGRAPH Asia 2023 Conference Proceedings},
                 year      = {2023}

Animated bunny
Animations over a static background are possible by detecting shadows on the input image as a whole and moving the 3D render.
Realistic shadow compositing must consider ground shadow interactions as well as real shadows cast on the virtual object.
Non-planar surfaces may cause the object to look less realistic if approximated as a plane. Our method can be extended to non-flat grounds given the 3D surface as input.
Noise in the background image can cause issues. Upon closer inspection, the noisy pixels in our shadow border are similar to the noisy pixels in the sand.
Unknown geometries are not accounted for unless 3D data is available (e.g., the bench close to the bunny's head does not make the shadow darker).
Unusual surface textures, like the orange ground above, are more challenging to generalize to, given the scarcity of training data available.
Ground textures with detailed shadow patterns are especially challenging to discern from unknown ground textures, causing innacuracies in shadow blending (e.g., floor gaps in the image above).