Gaussian Splatting: Papers #4

Here are the latest papers related to Gaussian Splatting! 🤘

FlowMap: High-Quality Camera Poses, Intrinsics, and Depth via Gradient Descent

by Cameron Smith, David Charatan, Ayush Tewari, Vincent Sitzmann
Published on 2024–04–23
Read the PDF // Project Website

This paper presents “FlowMap,” an end-to-end differentiable technique that accurately determines camera poses, camera intrinsics, and dense depth per video frame. Using a least-squares objective that compares depth-induced optical flow to off-the-shelf optical flow and point tracking correspondences, the method optimizes per-video via gradient descent. It introduces re-parameterizations of depth, intrinsics, and pose that facilitate first-order optimization. The results demonstrate that the method’s calculated camera settings and depth enable photorealistic novel view synthesis over 360-degree trajectories using Gaussian Splatting. It surpasses traditional bundle adjustment approaches and matches the performance of COLMAP, a leading SfM method, on novel view synthesis, offering a completely differentiable and innovative approach to structure from motion.

TalkingGaussian: Structure-Persistent 3D Talking Head Synthesis via Gaussian Splatting

by Jiahe Li, Jiawei Zhang, Xiao Bai, Jin Zheng, Xin Ning, Jun Zhou, Lin Gu
Published on 2024–04–23
Read the PDF // Github, project website, and video

“TalkingGaussian” introduces a novel framework for synthesizing 3D talking heads with high fidelity, employing a deformation-based approach within radiance fields. By using Gaussian Splatting, it modifies facial movements through smooth deformations to Gaussian primitives, avoiding the complexities of direct appearance change. This simplification allows for precise motion and feature preservation. The model also addresses face-mouth motion inconsistencies by separating the modeling of the face and mouth, thus simplifying learning tasks and enhancing the accuracy of mouth movements. The method outperforms previous models in producing lip-synchronized videos with better facial fidelity and efficiency.