Speaker
Fangneng Zhan is a postdoctoral researcher at Max Planck Institute for Informatics with Prof. Christian Theobalt. He received his Ph.D. degree of Computer Science & Engineering from Nanyang Technological University, Singapore in 2021. His research interests are Neural Rendering and Generative Models (e.g., NeRF, 3D GAN, Diffusion Models). Aligning with the dictum of “What I cannot create, I do not understand” by Richard Feynman, his research goals are (1) Developing algorithms to synthesize and reconstruct the visual world and (2) Exploring how visual analysis can benefit from the advance of visual synthesis.
Abstract
In recent years, neural fields have been a popular framework in neural rendering, especially with the emergence of NeRF for novel view synthesis. Aiming to boost the computation efficiency and rendering quality of 3D scenes, a popular line of research maps the 3D coordinate system to another measuring system, e.g., 2D manifolds and hash tables, for modeling neural fields. The conversion of coordinate systems can be typically dubbed as gauge transformation, which is usually a pre-defined mapping function, e.g., orthogonal projection or spatial hash function. This begs a question: can we directly learn a desired gauge transformation along with the neural field in an end-to-end manner? In this talk, we will share our work on developing end-to-end learning framework to jointly optimize the gauge transformation and neural fields. To counter the problem that the learning of gauge transformations can collapse easily, we also present the derivation of a general regularization mechanism from the principle of information conservation during the gauge transformation. The extensive experiments and thorough analysis show the superiority of learned gauge transformations. We hope this talk can inspire more insights for learning gauge transformations in rendering pipelines.