VGGT-Splat

Feed-forward VGGT poses vs. COLMAP for 3D Gaussian Splatting,
from 8 frames to all

Jiaming Chu

Independent research project, 2026

Question

VGGT predicts camera poses, a point cloud and per-frame depth for a set of images in a single forward pass, so it can replace COLMAP as the front end of 3D Gaussian Splatting. For a casual phone capture, two practical questions follow: how many frames does each pipeline need before it works, and with few frames, does supervising 3DGS with VGGT's depth recover the missing quality?

Setup

6 scenes: three self-captured phone videos (desk, lawn, room; 1080p, sampled at 3 fps, trained at 960×540) and three Mip-NeRF 360 scenes (garden, bicycle, counter). Each scene has a fixed test set (every 8th frame after removing blurry frames, as in Mip-NeRF 360) and nested training sets of N = 8 / 16 / 32 / all frames. 3 methods share the same gsplat trainer with per-camera pose refinement, stopped at 7k steps:

• COLMAP: COLMAP 4.2.1 with exhaustive matching, sparse points as initialization.
• VGGT: VGGT-1B poses from one forward pass, with the top 50% most confident points as initialization.
• VGGT + depth: the same, plus a Pearson loss between rendered inverse depth and VGGT depth (λ = 0.2).

Evaluation follows the protocol used for pose-free methods. Test cameras come from a joint run on training and test frames, are mapped into the trained model's frame by a Sim(3) fitted on the shared training cameras, and are refined for 500 steps with the Gaussians frozen. Methods are compared on the test frames they all registered. Re-running a configuration with different seeds moves PSNR by 0.06–0.35 dB (standard deviation), so single-run differences below about 0.3 dB are not conclusive; the depth-prior comparison is therefore checked with seed repeats.

Finding 1: with few frames, COLMAP fails and VGGT does not

COLMAP needs enough overlap between views to match features and register cameras. With 8 frames it fails outright on desk, room, bicycle and counter, and registers 3/8 and 6/8 frames on lawn and garden. With 16 frames it still fails on lawn, room and counter. Even at 32 frames it registers only 10/32 frames on bicycle and 15/32 on desk; on desk this is why its PSNR is lower than at 16. VGGT returns a pose for every frame in every configuration.

Finding 2: with all frames, VGGT is less precise

With every frame available, COLMAP leads by 2.1–4.6 dB on five scenes and by 0.7 dB on desk. To check whether this comes from VGGT covering less of the scene, I split each test image into pixels seen by at least one training view and pixels seen by none. On the seen pixels alone VGGT is still behind: garden 21.7 vs. 26.3 dB, bicycle 20.7 vs. 22.8, counter 24.9 vs. 27.2. The renders are smeared rather than incomplete. On lawn, garden and bicycle, VGGT's PSNR rises from 32 frames to all frames while LPIPS gets worse, the signature of blur. Likely, feed-forward poses are not mutually consistent enough for 3DGS to resolve fine detail, while bundle-adjusted COLMAP poses are.

VGGT's advantage at this end is speed: it estimates poses for all frames in 47–73 s, while COLMAP takes 167–790 s on the same GPU.

Finding 3: with few frames, the bottleneck is coverage

With 8 frames, large parts of each test view were never seen by any training view. Those pixels are 0.5–9 dB worse than the seen ones, and the fraction of unseen pixels in a test frame is strongly anti-correlated with that frame's PSNR. Most of the error is in unseen regions rather than seen ones, so coverage, not pose accuracy, is the main limit here.

Finding 4: a depth prior changes the geometry, not the novel views

The depth loss does what it is designed to do: on training views, the local correlation between rendered depth and VGGT depth rises from 0.4–0.8 to 0.7–0.9. But novel-view PSNR barely moves. After repeating the relevant configurations with three seeds, only two differences exceed noise: desk at 32 frames gets 0.68 dB worse and lawn at 8 frames gets 0.17 dB better. Finding 3 explains why. The prior constrains surfaces the training views already see, while the error is concentrated in regions they do not.

More scenes

All frames, COLMAP on the left and VGGT on the right. For the 360° scenes (garden above, bicycle, counter) both models are rendered along one elliptical orbit around the scene center: the orbit is built from COLMAP's training cameras and mapped into the VGGT model by a Sim(3) fitted on the shared training cameras (median center error under 1% of the trajectory scale), so both sides always show the same novel view. The self-captured scenes were not filmed all the way around, so they are shown along the training camera path instead. Desk shows both methods; lawn and room show VGGT only, because COLMAP registered fewer frames there (149/156 and 75/114) and its camera path differs.

All results

Test PSNR (dB) after test-time pose refinement, averaged over the test frames every usable method registered. Brackets give COLMAP's registered / total test frames when it did not register all of them; the other methods are averaged over the same subset. fail = pose estimation or alignment failed, or COLMAP's two reconstructions disagree. Best per column in bold.

Takeaway

For a casual capture of 16 frames or fewer, VGGT is the safer front end for 3D Gaussian Splatting: it always produces usable poses, and COLMAP often does not. Once a capture has a hundred frames or more, COLMAP's bundle adjustment gives noticeably sharper results on most scenes, at 3.6–13× the pose-estimation time. A VGGT depth prior is not a substitute for more views: with few frames, what is missing is coverage, and only more views or priors on unseen regions can supply it.

Limitations

One capture per scene and, for the main grid, one seed; seed repeats cover a subset of configurations. COLMAP is not deterministic with few frames: reproducing garden at 8 frames in a fresh environment, COLMAP registered 5/8 training frames instead of 6/8 and reached 13.35 dB instead of 14.27 (VGGT: 15.67 vs. 15.64 dB), although five further COLMAP runs on the same frames all registered 6/8. Training stops at 7k steps for every run (on desk at 16 frames, the 30k checkpoint was slightly worse on test views, with the same ranking). Whether a test pixel is “seen” by a training view is judged from the geometry each model reconstructs; that geometry may itself be inaccurate, so the split is only an approximation. Test cameras come from a separate joint run and are refined with the scene frozen, the standard protocol for pose-free methods but not the same as evaluating with ground-truth test poses.

Reproducibility

Code, the evaluation protocol, and the scripts that produce every table and figure on this page are in the repository, with pinned versions of gsplat, VGGT and COLMAP and the commands to rerun the full grid. The trained all-frames models of every scene (.splat and compressed .ply) and the self-captured frames are attached to the release page. The 3DGS trainer is a modified copy of gsplat's example trainer; every change is listed in trainer/NOTICE.md.

Citation

@misc{chu2026vggtsplat,
  title        = {VGGT-Splat: Feed-forward VGGT Poses vs. COLMAP for
                  3D Gaussian Splatting, from 8 Frames to All},
  author       = {Chu, Jiaming},
  year         = {2026},
  howpublished = {\url{https://github.com/chu-jiaming/VGGT-Splat}},
  note         = {Version 1.0.0}
}

← Jiaming Chu