to appear in CVPR 2016, Las Vegas, NV
Undoing the image formation process and therefore decomposing appearance into its intrinsic properties is a challenging task due to the under-constraint nature of this inverse problem. While significant progress has been made on inferring shape, materials and illumination from images only, progress in an unconstrained setting is still limited. We propose a convolutional neural architecture to estimate reflectance maps of specular materials in natural lighting conditions. We achieve this in an end-to-end learning formulation that directly predicts a reflectance map from the image itself. We show how to improve estimates by facilitating additional supervision in an indirect scheme that first predicts surface orientation and afterwards predicts the reflectance map by a learning-based sparse data interpolation.
In order to analyze performance on this difficult task, we propose a new challenge of Specular MAterials on SHapes with complex IllumiNation (SMASHINg) using both synthetic and real images. Furthermore, we show the application of our method to a range of image-based editing tasks on real images.
Here you can
Using the menu on top you can navigate in the different pages.
In the following video you can see material change and shape manipulation using the inferred Reflectance maps. Once our framework estimates the Reflectance maps for different images, we are able to transfer the material from one image to another. On the bottom, two input images are shown from which our approach estimates reflectance maps as visualized as spheres above the respective images. In the top half, estimated normals from the left input image are combined with the estimated reflectance maps of the right input image in order to synthesise a new car. We also show how the normal information can be altered via strokes performed by a user.
The online interactive demo illustrated in the video is under construction with three.js