Benchmark Results - E3D-Bench (original) (raw)

Benchmark Results - E3D-Bench

End-to-End 3D Geometric Foundation Models

1The University of Texas at Austin 2Brown University
3University of Central Florida 4NVIDIA Research 5Stanford University

Abstract

Spatial intelligence, encompassing 3D reconstruction, perception, and reasoning, is fundamental to applications such as robotics, aerial imaging, and extended reality. A key enabler is the real-time, accurate estimation of core 3D attributes (camera parameters, point clouds, depth maps, and 3D point tracks) from unstructured or streaming imagery. Inspired by the success of large foundation models in language and 2D vision, a new class of end-to-end 3D geometric foundation models (GFMs) has emerged, directly predicting dense 3D representations in a single feed-forward pass, eliminating the need for slow or unavailable precomputed camera parameters. Since late 2023, the field has exploded with diverse variants. With the rapid proliferation of 3D GFMs, we ask:

Q1 Can GFMs serve as an effective and robust foundation for diverse 3D tasks and scenarios?

Q2 Can GFMs serve as an efficient foundation, especially for latency-constrained 3D applications?

In this work, we present the first comprehensive benchmark for 3D GFMs, covering five core tasks: sparse-view depth estimation, video depth estimation, 3D reconstruction, multi-view pose estimation, novel view synthesis, and spanning both standard and challenging out-of-distribution datasets. Our standardized toolkit automates dataset handling, evaluation protocols, and metric computation to ensure fair, reproducible comparisons. We evaluate 16 state-of-the-art GFMs, revealing their strengths and limitations across tasks and domains, and derive key insights to guide future model scaling and optimization. All code, evaluation scripts, and processed data will be publicly released to accelerate research in 3D spatial AI.

Effectiveness

Extremely Sparse Dense

Object-Centric Indoor Scenes

In Distribution Long Sequence Street Driving Indoor-Outdoor Drone Air-Ground

Normalized Metric

Object-Centric Indoor Scene Outdoor Scene Mixed Scene

Normalized Metric

Indoor Scene Outdoor Scene Large Dynamic Motion Drone Scene Mixed Scene

Object-Centric Indoor Scenes Drone Scenes

Inference Efficiency

Findings and Takeaways

What Is the Impact of Tasks with Different Difficulties?

• Multi-view geometry inference is inherently harder than pair-view inference.
• Directly predicting dense 3D scene representations is much more challenging than estimating individual 3D attributes like depth and camera poses.
• Metric-scale depth estimation remains a key challenge for GFMs.
• Joint prediction of multiple geometric attributes (e.g., pose, depth, matching) may underlie recent performance gains.

Takeaway 1: Current GFMs are promising but face significant challenges when learning from overly complex tasks. Recommendation: Carefully decomposing difficult tasks (e.g., jointly predicting geometry, pose, depth, and tracking) into simpler sub-problems can facilitate more effective learning, especially under limited 3D data.

Do GFMs Generalize Well on Different Data Domains?

• GFMs struggle to generalize in domains with extreme data scarcity.

Takeaway 2: Diverse, high-quality data is critical for strong generalization. To improve robustness in underrepresented domains, GFMs must be trained on data that covers broader distributions and metric-scale annotations.

Hints for Model Architecture Design, ViT or Diffusion? Strong 2D Feature Extractor?

• No single design, feed-forward ViT or diffusion, is universally superior.
• Stronger 2D foundation models can significantly enhance 3D GFMs.

Takeaway 3: No single backbone—feed -forward ViT or diffusion, dominates; architecture choice should align with task needs. Moreover, leveraging strong 2D feature extractors (e.g., DINO) substantially boosts 3D performance.

Are Current GFMs Ready for Real-Time Perception Systems?

• Despite progress, GFMs still lack the efficiency required for real-time 3D applications.

Takeaway 4: As GFMs scale to handle more views and complex tasks, efficiency becomes as critical as accuracy for enabling real-time 3D perception.

Citation

@article{cong2025e3dbench,
  title={E3D-Bench: An End-to-End Benchmark for 3D Geometric Foundation Models},
  author={Cong, Wenyan and Liang, Yiqing and Zhang, Yancheng and Yang, Ziyi and Wang, Yan and Ivanovic, Boris and Pavone, Marco and Chen, Chen and Wang, Zhangyang and Fan, Zhiwen},
  journal={arXiv preprint arXiv:2506.01933},
  year={2025}
}