GECO🦎: Generative Image-to-3D within a SECOnd

Arxiv 2024


Chen Wang1, Jiatao Gu2, Xiaoxiao Long3, Yuan Liu3, Lingjie Liu1

1University of Pennsylvania    2Apple     3The University of Hong Kong     

Abstract


TL;DR: We propose a 3D distillation pipeline for feedforward 3D generation that operates in less than one second.

We integrated the distillation method in GECO for more advanced teachers (e.g. InstantMesh) to achieve better results.

3D generation has seen remarkable progress in recent years. Existing techniques, such as score distillation methods, produce notable results but require extensive per-scene optimization, impacting time efficiency. Alternatively, reconstruction-based approaches prioritize efficiency but compromise the quality due to their limited handling of uncertainty. We introduce GECO, a novel method for high-quality 3D generative modeling that operates within a second. Our approach addresses the prevalent issues of uncertainty and inefficiency in current methods through a two-stage approach. In the initial stage, we train a single-step multi-view generative model with score distillation. Then, a second-stage distillation is applied to address the challenge of view inconsistency from the multi-view prediction. This two-stage process ensures a balanced approach to 3D generation, optimizing both quality and efficiency. Our comprehensive experiments demonstrate that GECO achieves high-quality image-to-3D generation with an unprecedented level of efficiency.


Image to 3D


GECO can achieve great results on image-to-3D generation for in-the-wild data.


Comparison over Reconstruction-based Methods


GECO can handle the back-view of the object thanks to uncertainty modeling.


Interactive Meshes



Citation


@article{wang2024geco,
  title={GECO: Generative Image-to-3D within a Second},
  author={Wang, Chen and Gu, Jiatao and Long, Xiaoxiao and Liu, Yuan and Liu, Lingjie},
  journal={arXiv},
  year={2024}
}