Joint estimation of spectral illuminant and reflectance from an RGB image

Abstract

This paper addresses the challenge of accurately estimating both the full illuminant spectral power distribution (SPD) and the per-pixel spectral reflectance from an RGB image captured with a known camera. Full spectral information allows us to perform more accurate white balance, or render the scene under another illuminant. Traditional color constancy methods focus on predicting the illuminant color as a 3-dimensional projection of the infinite-dimensional illuminant SPD onto the camera spectral sensitivity functions (SSFs) space. Because different illuminants can have the same projection in the 3-dimensional SSFs space, those traditional methods cannot differentiate between such illuminants (metamers in the camera response space) and hence remove the color cast resulting from the illumination. We reconstruct the spectral information using a neural network with two interconnected branches: one branch predicts the illuminant SPD, and the other reconstructs the per-pixel spectral reflectance by integrating the predicted SPD within its intermediate layers. Experimental results demonstrate that our framework achieves superior performance in estimating both the illuminant SPD and the per-pixel spectral reflectance compared to the previous approach.