Synthetic rooms rendered from 3D-FRONT. Clean geometry, controlled lighting, unambiguous layout.
HARMONY
Hierarchical Agentic Reasoning for MONocular Image-to-Scene Synthesis
University of Pennsylvania
Hierarchical Agentic Reasoning for MONocular Image-to-Scene Synthesis
Compositional 3D scene reconstruction has recently been explored from two directions: agentic reasoning that provides semantic understanding of spatial relationships but lacks precise alignment with input images; and visual geometry foundation models that predict dense point maps from input images but the reconstruction quality is limited. Therefore, recovering a complete 3D scene from a single monocular image with accurate inter-object relationships and high-fidelity reconstruction quality remains challenging. In this paper, we present HARMONY, a hierarchical chain-of-thought framework that leverages both agentic reasoning and visual geometry foundation. Given an indoor image, starting from an empty 3D floorplan, HARMONY first calibrates the camera against the reference image to establish a semantically-grounded spatial frame, then uses agentic VLM reasoning to recover the 3D room layout and an initial placement order. It then places the objects in a hierarchical order, from wall-mounted elements, free-standing furniture, to dependent decorations on top of furniture. We also use depth-first traversal for furniture so each placement conditions on previously resolved structure and a reflective feedback loop to avoid error accumulation. After VLM placement, we use the point cloud estimations to perform geometry-based refinement so that the rendered image aligns better with the input. By applying even open-source VLMs, HARMONY can produce 3D scenes that are semantically consistent and perceptually aligned with the reference image, extending single-image compositional reconstruction to complex indoor scene images. Experiments on challenging real-world input images demonstrate that our method can produce a high-quality single image to 3D compositional reconstruction.
A hierarchical chain-of-thought framework: ground the VLM in 3D, place objects stage by stage, then refine with visual geometry.
Pipeline overview. HARMONY creates an empty 3D room and anchors the camera from the estimated point cloud and a Manhattan frame, segments and reconstructs each object into a 3D mesh, then uses a VLM to plan and place objects in three stages, using point clouds for correction.
The VLM estimates room dimensions from the image, anchors to the deepest visible corner, and labels the walls. An anchor-based pinning procedure with a Manhattan frame calibrates the render camera to the input view.
LocateAnything with SAM segments each object; the VLM inpaints occluded regions and verifies semantic consistency. Meshes are reconstructed, canonicalized via PCA, and labeled with a facing direction.
Objects are placed in a depth-first order across three tiers: wall-mounted, free-standing furniture, then decorations — so each new object appears in a clean, unoccluded render for reasoning about orientation and pairing.
Silhouette and point-cloud alignment fix precise position and scale. After each stage, a reflective feedback loop has the VLM critique the render against the input to catch missing items, wrong scales, and bad orientations.
Select a scene, orbit to explore, click any object to view its specs, and drag furniture to move it around the room.
Meshes are compressed & texture-downsampled for fast web loading, so quality is slightly below the originals.
Six single-photo reconstructions, assembled into one apartment from a generated floorplan.
Each room is a separate HARMONY reconstruction from the photo above; the floorplan and door placement are proposed by the VLM from the room list alone. The live scene reads like the cutaway render beside it — ceilings removed and wall slabs drawn half-transparent — and a room photo flies the camera to the viewpoint that photo was taken from.
300 indoor scenes across three complexity tiers, for evaluating agentic single-image to 3D scene generation.
Each case pairs a reference photograph with a reconstructed scene, a physically-lit Cycles render and the camera used to produce it — so a method can be scored from the input viewpoint without re-solving the camera. The tiers separate what is hard about a room: synthetic layouts, real photographs of a single dominant layout, and real photographs of cluttered multi-object interiors.
Synthetic rooms rendered from 3D-FRONT. Clean geometry, controlled lighting, unambiguous layout.
Real photographs with a single dominant layout — real materials and lighting, but one clear arrangement to recover.
Real photographs of cluttered, multi-object interiors — occlusion, dense decoration and competing layouts.
input.*reference photograph — the model inputrender_blender.pngphysically-lit Cycles render of the reconstructionscene_full.glbreconstructed scene — shell, furniture, decorationscamera_vggt.jsoncamera used for the render, for input-viewpoint scoringReleased under CC-BY-NC-4.0 — the most restrictive constituent license applied uniformly. Input images stay under their original terms: Pexels (196), 3D-FRONT (100), Unsplash (2) and 3D-FUTURE (2); per-scene sources are listed in attribution.csv. The 3D-FRONT easy split is research-only and requires citing 3D-FRONT.
Renders from the input viewpoint — HARMONY vs. baselines.
@article{sun2026harmony,
title = {HARMONY: Hierarchical Agentic Reasoning for Monocular Image-to-Scene Synthesis},
author = {Sun, Shufan and Wang, Chen and Gu, Jiatao and Liu, Lingjie},
journal = {arXiv preprint},
year = {2026}
}