[1912.02314v1] Computational Mirrors: Blind Inverse Light Transport by Deep Matrix Factorization
We find it remarkable that merely asking for latent factors easily expressible by a CNN is sufficient to solve our problem, allowing us to entirely bypass challenges such as the estimation of the geometry and reflectance properties of the scene

Abstract We recover a video of the motion taking place in a hidden scene by observing changes in indirect illumination in a nearby uncalibrated visible region. We solve this problem by factoring the observed video into a matrix product between the unknown hidden scene video and an unknown light transport matrix. This task is extremely ill-posed, as any non-negative factorization will satisfy the data. Inspired by recent work on the Deep Image Prior, we parameterize the factor matrices using randomly initialized convolutional neural networks trained in a one-off manner, and show that this results in decompositions that reflect the true motion in the hidden scene.
‹Figure 1: A typical experimental setup used throughout this paper. A camera views a pile of clutter, while a hidden video L is being projected outside the direct view Z of the camera. We wish to recover the hidden video from the shadows and shading observed on the clutter. The room lights in this photograph were turned on for the purpose of visualization only. During regular capture, we try to minimize any sources of ambient light. We encourage the reader to view the supplemental video to see the data and our results in motion. (Inverse Light Transport)Figure 2: Reconstructions with known light transport matrix. (Inversion with Known Light Transport Matrix)Figure 3: High level overview of our matrix factorization approach. The CNNs are initialized randomly and “overfitted” to map two vectors of noise onto two matrices T and L, with the goal of making their product match the input matrix Z. In contrast to optimizing directly for the entries of T and L, this procedure regularizes the factorization to prefer image-like structure in these matrices. (Method)Figure 4: Matrix factorization results. The input to the method is a product of the two leftmost matrices. Our method finds visually readable factors, and e.g. recovers all three of the faint curves on the first example. On the right, we show two different baselines: one computed with Matlab’s non-negative matrix factorization (in alternating least squares mode), and one using our code but optimizing directly on matrix entries instead of using the CNN, with an L1 smoothness prior. (Experiments and Results)Figure 5: An overview of the architecture and data flow of our blind inverse light transport method. Also shown (bottom left) are examples of the left singular vectors stored in U. L and Q are convolutional neural networks, and the remainder of the blocks are either multidimensional tensors or matrices, with dimensions shown at the edges. The matrices in the shaded region are computed once during initialization. The input Z to the method is shown in the lower right corner. (Method)Figure 6: Blind light transport factorization using our method. The first three sequences are projected onto a wall behind the camera. The Lego sequence is performed live in front of the illuminated wall. (Experiments and Results)›