Soumyadip Sengupta (Roni)

(he/him)

I am a Postdoctoral Research Associate in Computer Science & Engineering at University of Washington. I am working with Prof. Steve Seitz, Prof. Brian Curless and Prof. Ira Kemelmacher-Shlizerman in the UW Reality Lab and GRAIL.

I will be joining the University of North Carolina at Chapel Hill as an Assistant Professor of Computer Science starting Fall 2022

I completed my Ph.D. (Aug 2013 - Jan 2019) from University of Maryland, College Park (UMD), advised by Prof. David Jacobs and my undergraduate degree (2009-2013) in ECE from Jadavpur University, Kolkata, India.

I also had the pleasure to work with:

Jinwei Gu, Kihwan Kim, Guilin Liu and Jan Kautz at NVIDIA Research (April-Nov 2018)
Linjie Luo and Chen Cao at Snapchat Research (May-Aug 2017)
Prof. Tom Goldstein at UMD (2015-2017)
Prof. Ronen Basri at Faculty of Mathematics and Computer Science, The Weizmann Institute of Science, Israel (June 2015 - June 2016)
Prof. Günter Rudolph at TU Dortmund, Germany, under DAAD schoalrship (May-Aug 2012).

Contact

Email: soumya91 at cs.washington.edu
Follow @SenguptRoni
Website: Google Scholar
Resume/CV: CV
Office: CSE2 233, The Bill & Melinda Gates Center
Mail: Paul G. Allen School of Computer Science & Engineering,
University of Washington, Seattle

Research Interest

The goal of my research is to create next-generation video communication and content creation by democratizing high-quality video production and editing. To achieve this goal, I develop algorithms at the intersection of Computer Vision, Graphics, and Machine Learning that can edit various components of an image or a video by understanding its intrinsic components. My research forms new problem statements on personalized AI models and creates novel strategies with minimal user interaction. This will produce high-quality videos by better constraining the intrinsic decomposition process.

Thesis

Constraints and Priors for Inverse Rendering from Limited Observations
Soumyadip Sengupta
Doctoral Thesis, University of Maryland, January 2019
[pdf]

Selected Publications

Computer Vision and Machine Learning:

	Real-Time Light-Weight Near-Field Photometric Stereo Daniel Lichy, Soumyadip Sengupta, David Jacobs (CVPR 2022) [Paper][Project Page][Code] Near-field Photometric Stereo technique is useful for 3D imaging of large objects. We capture multiple images of an object by moving a flashlight and reconstruct the 3D mesh. Our method is significnatly faster and memory-efficient while producing better quality than SOTA methods.
	Robust High-Resolution Video Matting with Temporal Guidance Peter Lin, Linjie Yang, Imran Saleemi, Soumyadip Sengupta (WACV 2022) [Paper][Project Page][Code] Background Removal a.k.a Alpha matting on videos by exploiting temporal information with a recurrent architecture. Does not require capturing background image or manual annotations.
	A Light Stage on Every Desk Soumyadip Sengupta, Brian Curless, Ira Kemelmacher-Shlizerman, Steve Seitz (ICCV 2021) [Paper][Project Page] We learn a personalized relighting model by capturing a person watching YouTube videos. Potential application includes relighting during a zoom call.
	Shape and Material Capture at Home Daniel Lichy, Jiaye Wu, Soumyadip Sengupta, David Jacobs (CVPR 2021) [Paper][Project Page][Code] High-quality Photometric Stereo can be achieved with a simple flashlight. Recovers hi-res geometry and reflectance by progressively refining the predictions at each scale, conditioned on the prediction at previous scale.
	Real-Time High Resolution Background Matting Peter Lin, Andrey Ryabtsev, Soumyadip Sengupta, Brian Curless, Steve Seitz, Ira Kemelmacher-Shlizerman (CVPR 2021 ORAL)(Best Paper Candidate 32/1600+ accepted papers)(Best Student Paper Honorable Mentions) [Paper][Project Page][Code] Background replacement at 30fps on 4K and 60fps on HD. Alpha matte is first extracted at low-res and then selectively refined with patches.
	Lifespan Age Transformation Synthesis Roy Or-El, Soumyadip Sengupta, Ohad Fried, Eli Shechtman, Ira Kemelmacher-Shlizerman (ECCV 2020) [Paper][Project Page][Code] Age transformation from 0-70. Continuous aging is modeled by assuming 10 anchor age classes with interpolation in the latent space between them.
	Background Matting: The World is Your Green Screen Soumyadip Sengupta, Vivek Jayaram, Brian Curless, Steve Seitz, Ira Kemelmacher-Shlizerman (CVPR 2020) [Paper][Project Page][Code][Two Minute Papers Video][Microsoft AI using our code][CEO of Microsoft Satya Nadella talks about our work] By simply capturing an additional image of the background, alpha matte can be extracted easily without requiring extensive human annotation in form of trimap.
	Neural Inverse Rendering of an Indoor Scene from a Single Image Soumyadip Sengupta, Jinwei Gu, Kihwan Kim, Guilin Liu, David Jacobs, Jan Kautz (ICCV 2019) [Paper][Project Page] Self-supervision on real data is achieved with a Residual Appearnace Renderer network. It can cast shadows, add inter-reflections and near-field lighting, given the normal and albedo of the scene.
	SfSNet : Learning Shape, Reflectance and Illuminance of Faces in the Wild Soumyadip Sengupta, Angjoo Kanazawa, Carlos D. Castillo, David Jacobs. CVPR 2018 [Spotlight]. [Paper] [Project Page / Code] Decomposes an unconstrained human face into surface normal, albedo and spherical harmonics lighting. Learns from synthetic 3DMM followed by self-supervised finetuning on unlabelled real images. Soumyadip Sengupta, Daniel Lichy, Angjoo Kanazawa, Carlos D. Castillo, David Jacobs. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI) 2020. [Paper] Introduces SfSMesh that utilizes the surface normal predicted by SfSNet to reconstruct a 3D face mesh.
	A New Rank Constraint on Multi-view Fundamental Matrices, and its Application to Camera Location Recovery Soumyadip Sengupta, Tal Amir, Meirav Galun, Amit Singer, Tom Goldstein, David Jacobs, Ronen Basri. CVPR 2017 [Spotlight]. [Paper] [Code] We prove that a matrix formed by stacking fundamental matrices between pairs of images has rank 6. We then introduce a non-linear optimization algorithm based on ADMM, that can better estimate the camera parameters using this rank constraint. This improves Structure-from-Motion algorithms which require initial camera estimation (bundle adjustment).
	Solving Uncalibrated Photometric Stereo Using Fewer Images by Jointly Optimizing Low-rank Matrix Completion and Integrability Soumyadip Sengupta, Hao Zhou, Walter Forkel, Ronen Basri, Tom Goldstein, David Jacobs. Journal of Mathematical Imaging and Vision (JMIV), 2018. [Paper] We solve uncalibrated Photometric Stereo using as few as 4-6 images as a rank-constrained non-linear optimization with ADMM.
	Frontal to Profile Face Verification in the Wild Soumyadip Sengupta, Jun-Cheng Chen, Carlos D. Castillo, Vishal M. Patel, Rama Chellappa, David Jacobs. WACV 2016. [Project Page] , [Paper] We introduce a dataset of frontal vs profile face verfication in the wild -- CFP. We show that SOTA face verification algorithms degrade about 10% on frontal-profile verification compared to frontal-frontal. Our dataset has been widely used to improve face verification across poses, but also for face warping and pose synthesis with GAN.
	A Frequency Domain Approach to Silhouette Based Gait Recognition Soumyadip Sengupta, Udit Halder, Rameshwar Panda, Ananda S Chowdhury. NCVPRIPG 2013. [Paper]

Multi-objective Optimization:

“An evolutionary multiobjective sleep-scheduling scheme for differentiated coverage in wireless sensor networks”, Soumyadip Sengupta, Swagatam Das, Md Nasir, AV Vasilakos, Witold Pedrycz, IEEE Transactions Systems, Men and Cybernetics - Part C, 2012. [Paper]
“A dynamic neighborhood learning based particle swarm optimizer for global numerical optimization”, Md Nasir, Swagatam Das, Dipankar Maity, Soumyadip Sengupta, Udit Halder, PN Suganthan, Elsevier Information Sciences, 2012. [Paper]
“Evenly spaced pareto front approximations for tricriteria problems based on triangulation”, Günter Rudolph, Heike Trautmann, Soumyadip Sengupta, Oliver Schütze, Evolutionary Multi-Criterion Optimization, 2013.[Paper]

Teaching

Teaching Assistant: ENEE 222 Spring 2014 : Elements of Discrete Signal Analysis
Teaching Assistant: ENEE 420 Fall 2013 : Communications Systems