PoseGuard-NeRF

PoseGuard-NeRF

When does joint camera-pose optimization help NeRF,
and when does it just move the world?

Independent research project, 2026

Lego with 5% translation noise on every training camera. Left: fixed cameras, 14.1 dB test PSNR. Right: joint pose optimization with a camera learning-rate warm-up, 22.7 dB. Same data, same 30k steps.

TL;DR

Small pose errors wreck fixed-camera NeRF. Test PSNR falls from 27.1 dB to 14.1 dB at 5% translation noise and to 12.0 dB at 5° rotation noise.
Joint pose optimization recovers much of it. A camera learning-rate warm-up makes the naive version better in 9/9 conditions (+4.3 dB on average; 14.1 → 22.7 dB at 5% translation noise).
But turning on Naive pose optimization from perfect poses lowers PSNR from 27.1 to 17.9 dB. The reconstruction drifts as a whole: a Sim(3) fitted only on training camera centers, with the network frozen, recovers +5.5 dB (Naive) and +6.1 dB (CamP-style). Raw test metrics conflate scene quality with global coordinate drift.

Question

Real capture pipelines never give NeRF perfect camera poses, so a common remedy is to optimize per-camera pose corrections jointly with the radiance field. This raises two questions that are usually answered with a single test-PSNR number: how much does joint optimization help under a given amount of pose noise, and when test PSNR goes down, is the reconstruction actually worse, or is it a good reconstruction sitting in a slightly different coordinate frame than the held-out test cameras?

Setup

All runs use Nerfstudio's vanilla-NeRF on the Lego scene, trained for 30k steps. I perturb the training cameras at 9 noise levels (no noise; translation 0.5 / 1 / 2 / 5%; rotation 0.5 / 1 / 2 / 5°) and train 5 methods on each, for 45 runs in total:

Fixed cameras: the noisy poses are used as given.
Naive: a 6-D SO(3) × R³ correction per training camera, optimized jointly with the field.
CamP-style: the same correction in a preconditioned space that whitens each camera's projection Jacobian, re-implemented following CamP.
Naive + warm-up and CamP-style + warm-up: the camera learning rate ramps from 1e-8 to 1e-3 over the first 2,500 steps instead of starting at 1e-3.

Every run is followed by a pose audit and a Sim(3) alignment diagnostic: fit a similarity transform between ground-truth and optimized training camera centers, apply it to the test cameras, and re-render with the network frozen. No test image is used for fitting and no weight is updated, so any gain comes purely from undoing a global change of frame.

Renders of one test view for all five methods under 5% translation and 5 degree rotation noise — One test view for all five methods under 5% translation and 5° rotation noise.

Finding 1: pose noise breaks fixed cameras; joint optimization recovers

Fixed-camera NeRF is surprisingly fragile: test PSNR drops from 27.1 dB with perfect poses to 14.1 dB at 5% translation noise and 12.0 dB at 5° rotation noise. Joint pose optimization recovers much of it. Fixed cameras win only with perfect poses or 0.5% translation noise; in the other seven noisy conditions Naive + warm-up is best in six and 0.09 dB behind CamP-style + warm-up in the seventh (2° rotation).

The recovered cameras show why. Joint optimization pulls the noisy training cameras back to within 0.025 scene units on average (from 0.16); removing one global similarity transform brings that to 0.016.

Heatmap of test PSNR for 9 pose conditions by 5 methods — Test PSNR for all 45 runs (9 pose conditions × 5 methods).

Training-camera center errors before and after joint pose optimization, magnified 10x — Training-camera center errors before and after joint pose optimization (magnified 10×).

Finding 2: a camera learning-rate warm-up matters

Starting the camera learning rate near zero and ramping it up over 2,500 steps makes Naive pose optimization better in 9/9 pose conditions, by +4.3 dB on average. At 5% translation noise it lifts test PSNR from 14.1 dB (fixed cameras) to 22.7 dB. For CamP-style, warm-up helps in 8/9 conditions (+1.6 dB on average; 5% translation is the exception). Read in the aligned frame, though, the picture changes: on the GT condition the Naive gain is still +3.0 dB, while the CamP-style gain disappears (−0.06 dB).

Finding 3: most of the “drop from perfect poses” is coordinate drift

The counter-intuitive case is the GT condition. Enabling Naive pose optimization from perfect poses lowers test PSNR from 27.1 dB to 17.9 dB, which naively reads as “pose optimization damages the scene.” The renders say otherwise: they are sharp but shifted, so the error concentrates on object edges. Re-rendering the test cameras after a Sim(3) fitted on training camera centers only, with the network frozen, recovers +5.5 dB for Naive and +6.1 dB for CamP-style.

Ground truth, fixed-camera, naive raw and naive Sim(3)-aligned renders with per-pixel error maps — Same checkpoint, same test camera. PSNR values in this figure are for this *single test view*; the table below reports the full 200-view evaluation.

Method (GT condition)	Raw PSNR	Sim(3)-aligned PSNR	Gain
Fixed cameras (identity)	27.12	27.12	0.00
Naive	17.91	23.39	+5.48
CamP-style	19.90	26.04	+6.14
Naive + warm-up	22.71	26.42	+3.71
CamP-style + warm-up	21.08	25.98	+4.90

The pose diagnostics logged during training tell the same story: raw camera-center error plateaus, while the error left after removing one global similarity transform keeps shrinking. The scene is being reconstructed well; the world it lives in is moving.

Raw versus Sim(3)-aligned camera-center error over training — Raw vs. Sim(3)-aligned training-camera center error over training.

Takeaway

When camera poses are optimized, raw novel-view metrics measure two things at once: reconstruction quality and global coordinate drift relative to the test cameras. Reporting aligned metrics next to raw ones, using only training cameras to fit the alignment, separates the two.

Limitations

One scene (Lego), one seed (42).
The warm-up changes the whole camera learning-rate schedule, so the early phase alone is not isolated.
Full aligned rendering covers the GT condition only.

Reproducibility

Code, configs, metrics and poses for all 45 runs are in the repository (MIT license). CPU tests run in CI. The seven final checkpoints behind the alignment table and the comparison video are published on the release page. Training, resuming, auditing and aligned rendering need a CUDA GPU.

Citation

@misc{chu2026poseguardnerf,
  title        = {PoseGuard-NeRF: When Does Joint Camera-Pose Optimization
                  Help NeRF, and When Does It Just Move the World?},
  author       = {Chu, Jiaming},
  year         = {2026},
  howpublished = {\url{https://github.com/chu-jiaming/PoseGuard-NeRF}},
  note         = {Version 1.0.0}
}