Data-efficient Neural Appearance Manipulations

Abstract

Appearance manipulation techniques are indispensable for a wide range of applications that seek to create immersive and interactive experiences, including filmmaking, aug- mented/virtual reality (AR/VR), product design, and advertising. These techniques involve adjustments to lighting, material properties, textures, and geometry, fundamentally shaping how objects are perceived in a 3D scene. The demand for precise control over these scene components is increasing, as we come closer to the creation of virtual worlds that are indistinguishable from the real world. Traditional appearance editing techniques include artist-designed manipulations and, despite their effectiveness, require significant time and expertise. Consequently, machine learning (ML)-based approaches have emerged as key to the success of faster and accurate manipulations. This dissertation explores the potential of data-efficient learning-based techniques for manipulating three core aspects of appearance: fine details, transient attributes, and reflectance. It introduces two novel contributions: (1) an ML-based image map representation designed for fine detail editing in photographs, and (2) HyperBRDF, a bidirectional reflectance distribution function (BRDF) representation that enables sparse reconstruction, compression, and editing. By leveraging the known or learned priors that include problem-specific information, the proposed methods address the challenge of producing high-quality and visually appealing results in data-scarce regimes and generalise well to new inputs, offering photorealistic manipulations. Moving beyond image-based editing techniques, this dissertation further investigates appearance at the 3D scene level by combining physics-based principles with modern ML techniques, thereby pushing the boundaries of scene-level appearance control and physics based rendering systems, offering practical solutions for applications in visual effects, AR/VR, and beyond.