A Deep Perceptual Measure for Lens and Camera Calibration

Image editing and compositing have become ubiquitous in entertainment, from digital art to AR and VR experiences. To produce beautiful composites, the camera needs to be geometrically calibrated, which can be tedious and requires a physical calibration target. In place of the traditional multi-image calibration process, we propose to infer the camera calibration parameters such as pitch, roll, field of view, and lens distortion directly from a single image using a deep convolutional neural network. We train this network using automatically generated samples from a large-scale panorama dataset, yielding competitive accuracy in terms of standard `2 error. However, we argue that minimizing such standard error metrics might not be optimal for many applications. In this work, we investigate human sensitivity to inaccuracies in geometric camera calibration. To this end, we conduct a large-scale human perception study where we ask participants to judge the realism of 3D objects composited with correct and biased camera calibration parameters. Based on this study, we develop a new perceptual measure for camera calibration and demonstrate that our deep calibration network outperforms previous single-image based calibration methods both on standard metrics as well as on this novel perceptual measure. Finally, we demonstrate the use of our calibration network for several applications, including virtual object insertion, image retrieval, and compositing.

Paper (TPAMI 2023 version)

Yannick Hold-Geoffroy, Dominique Piché-Meunier, Kalyan Sunkavalli, Jean-Charles Bazin, François Rameau, and Jean-François Lalonde
A Deep Perceptual Measure for Lens and Camera Calibration
IEEE Transactions on Pattern Analysis and Machine Intelligence, to appear, 2023
[arXiv pre-print]

Supplementary material

We provide additional results in this supplementary document.

Live demo

We have two different versions of our most recent (TPAMI 2023) approach:

Paper (CVPR 2018 version)

Yannick Hold-Geoffroy, Kalyan Sunkavalli, Jonathan Eisenmann, Matt Fisher, Emiliano Gambaretto, Sunil Hadap, and Jean-François Lalonde
A Perceptual Measure for Deep Single Image Camera Calibration
IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
[arXiv pre-print]

Supplementary material

We provide additional results in this supplementary page.

Poster (CVPR 2018)

You can download the poster presented at CVPR 2018 in PDF format.


The authors gratefully acknowledge the following funding sources:

  • NSERC Discovery GRANT RGPIN-2014-05314
  • Korea NRF grant NRF-2017R1C1B5077030
  • FRQ-NT Ph.D. scholarship to Yannick Hold-Geoffroy
  • NSERC USRA to Dominique Piché-Meunier
  • A generous donation from Adobe to J-F Lalonde and J-C Bazin
  • NVIDIA Corporation with the donation of the Tesla K40 and Titan X GPUs used for this research.
  • REPARTI Strategic Network

Ulaval logo
Adobe logo