Deep Generative Prior

A/Prof. Chen Change Loy

Learning image prior is essential to solving various tasks of image restoration and manipulation, such as image colorization, image inpainting, and super-resolution. In the past decades, many image priors have been proposed to capture statistics of natural images. Despite their successes, these priors often serve a dedicated purpose. In this talk, I will share our efforts of leveraging Generative Adversarial Networks (GANs) that are trained on large-scale natural images for richer priors. Deep generative prior (DGP) offers compelling results in restoring missing semantics, e.g., color, patch, resolution, in degraded images. It also enables various image manipulation, including random jittering and image morphing. I will further show the possibility of mining 3D geometric clues from an off-the-shelf 2D GAN that is trained on RGB images only. We found that a pre-trained GAN indeed contains rich 3D knowledge and thus can be used to recover 3D shape from a single 2D image in an unsupervised manner. The recovered 3D shapes immediately allow high-quality image editing like relighting and object rotation. Lastly, I will discuss the way to overcome the efficiency bottleneck in GAN inversion so that we can use a pre-trained GAN for efficient and effective large-factor image super-resolution.


Chen Change Loy is a Nanyang Associate Professor with the School of Computer Science and Engineering, Nanyang Technological University, Singapore. He is also an Adjunct Associate Professor at the Chinese University of Hong Kong.

He received his PhD (2010) in Computer Science from the Queen Mary University of London. Before joining NTU, he served as a Research Assistant Professor at the MMLab of the Chinese University of Hong Kong, from 2013 to 2018. He is the recipient of the 2019 Nanyang Associate Professorship (Early Career Award) from Nanyang Technological University.

He is recognized by inclusion in the AI 2000 Most Influential Scholar Annual List (AI 2000). His research interests include computer vision and deep learning with a focus on image/video restoration, enhancement, and manipulation. His journal paper on image super-resolution was selected as the `Most Popular Article’ by TPAMI in 2016. It remains as one of the top 10 articles to date. He serves as an Associate Editor of IJCV and TPAMI. He also serves/served as the Area Chair of CVPR 2021, CVPR 2019, BMVC 2019, ECCV 2018, and BMVC 2018. He has co-organized several workshops and challenges at major computer vision conferences.