DSSG: Sparse Light Field-Guided 3D Gaussian Relighting with Depth-Semantic Optimization

High-Quality Relightable 3D Scene Reconstruction

Y. Huang • Y. Xu • X. Zhou • C. Pan* • S. Wang*

Beijing Forestry University • Beijing University of Aeronautics and Astronautics

Abstract

3D Gaussian relighting is essential for interactive object editing and photorealistic rendering. Current methods depend on dense SfM point clouds for BRDF initialization, but non-uniform distributions in complex material regions yield incomplete geometry and compromise light propagation in occluded areas, causing material decomposition ambiguities and unstable shadows. We propose DSSG, a relighting framework integrating depth-semantic optimization with sparse light field guidance. Our approach achieves physically plausible relighting through: (1) Sparse Large Variance (SLV) strategy for comprehensive light interaction coverage, (2) depth-semantic optimization using DPT and DINO-ViT for geometric consistency and cross-view semantic alignment, and (3) geometry-material alternating strategy for dynamic constraint scheduling. Experiments demonstrate improved novel view synthesis and relighting quality with reduced shadow errors under various lighting conditions.

Key Contributions

🎯 Depth-Semantic Optimization: Novel dual-constraint mechanism using DPT depth maps and DINO-ViT features to resolve cross-view diffuse inconsistencies and ensure material consistency.
🌟 SLV-based Initialization: Sparse Large Variance strategy with progressive frequency filtering for robust BRDF estimation without SfM dependency.
⚡ DSSG Framework: Complete framework achieving real-time relighting with high quality through collaborative geometry-material-lighting optimization.

Method Overview

Our DSSG framework comprises three stages: (1) SLV initialization for uniform Gaussian point cloud generation within camera frustums; (2) depth-semantic optimization using DPT-based depth maps and DINO-ViT features for geometric consistency and material alignment; (3) differentiable rendering for physically realistic dynamic relighting.

Results Gallery

Material Decomposition Results

Multi-Environment Material Decomposition

Material decomposition results under various lighting conditions, demonstrating the robustness of our method across different environments.

Quantitative Results

Method	PSNR ↑	SSIM ↑	LPIPS ↓	Relighting PSNR ↑
DSSG (Ours)	37.05	0.983	0.030	31.42
R3DG	36.80	0.982	0.028	31.00
TensorIR	35.80	0.978	0.049	29.69
InvRender	30.74	0.953	0.086	28.67

                🏆 Performance Highlights: DSSG achieves state-of-the-art performance with 37.05 dB PSNR in view synthesis and 31.42 dB in relighting tasks, demonstrating superior material decomposition and shadow quality.
            

Ablation Study

Select different components from the dropdowns below to compare results using the interactive sliders.

Left Image:

Right Image:

Ground Truth

Full DSSG

The Impact of Semantic Constraints

after removing semantic constraints, the material appearance of the balloon is significantly degraded, and the shadow quality also deteriorates.

Left Image:

Right Image:

Without Depth

Without Both

The Contribution of Depth Constraints to Geometric Reconstruction

Depthconstraints reduce geometricuncer taintyatobjectboundaries