Deep Multi-view Stereo for GTSFM

Abstract

Current structure-from-motion (SFM) pipelines integrated with multi-view stereo (MVS) module often applies traditional MVS algorithms. These algorithms cannot be well parallelized and can be slow when the data size and the resolution increase. For view synthesis, there are seldom SFM pipelines integrating it. This thesis focuses on how to integrate MVS and view synthesis efficiently into SFM pipelines, especially for the latest deep learning approaches. I first do a thorough survey in both domains, compare the advantages and disadvantages of the latest studies, and select the best-fit approach for our distributed SFM pipeline, Georgia Tech Structure from Motion (GTSFM). We implement the deep multi-view optimizer with PatchmatchNet and integrate it into the working graph of GTSFM. We also design an algorithm to boost a novel deep view synthesis algorithm, Instant-NGP, by forcing the reconstruction region on the overlapping Field-of-Views. This also enables us to extract high-quality dense polygon meshes of foreground objects directly from the reconstructed depth field. Experiments run on DTU and Skydio Crane Mast datasets suggest our MVS approach is more efficient than some popular SFM pipelines with MVS implemented.