A new research breakthrough demonstrates how AI coding agents can transform single photographs into fully executable 3D scenes, though the technology reveals a surprising blind spot in AI's self-assessment capabilities.

The system, called LEGO-Anything, employs an innovative "Image-to-Code" methodology where AI agents analyze photographs and write executable Blender programs to recreate the scenes. Unlike traditional 3D reconstruction methods that generate scenes directly, this approach produces actual code that explicitly captures objects, geometry, layout, and camera positioning, making the results fully editable and queryable like any software program.

The iterative process involves the AI agent writing initial code, executing it in Blender, evaluating the visual output against the original photograph, and making successive refinements. This code-based approach offers significant advantages, as the resulting programs can be run, checked, and modified using standard software development practices.

To properly evaluate these systems, researchers developed LEGO-Bench, a comprehensive benchmark containing 208 images from 104 indoor and outdoor scenes utilizing 443 registered assets. The benchmark addresses a fundamental challenge in 3D reconstruction evaluation: real photographs lack precise 3D ground truth for comparison, while simple synthetic scenes appear unrealistic. LEGO-Bench bridges this gap by rendering images from professionally constructed simulator scenes, providing natural-looking inputs while maintaining exact geometric data for automated scoring.

The benchmark evaluates reconstructions across three critical dimensions: validity (whether a usable scene was produced), geometric accuracy (precision of visible geometry), and visual similarity (how closely the appearance matches the original through pixel-by-pixel comparison of re-rendered scenes).

Testing across six GPT configurations revealed significant performance variations. GPT-6 Astra emerged as the clear leader, achieving 53.4% accuracy on indoor scenes and 39.6% on outdoor environments, while weaker configurations scored around 15%. The results demonstrated that scene complexity directly impacts accuracy, with outdoor environments proving more challenging than interior spaces.

A particularly interesting finding emerged when researchers increased the models' reasoning budget. GPT-6 variants showed substantial improvements, with Astra's performance on office scenes jumping from 32.3% to 61.8%. This suggests that given more computational resources for deliberation, these systems can achieve significantly better results.

However, the research uncovered a critical limitation: AI agents demonstrate remarkably poor self-assessment capabilities. When models were asked to judge which of two scene versions better matched the original photograph, their geometric evaluations performed at or below chance level. This means agents cannot reliably determine whether their own reconstructions have improved, leading to situations where they actually degrade their work during refinement attempts.

This discovery led to the development of LEGO-Plugin, an extension requiring no additional training that addresses these self-assessment failures. The plugin anchors initial scenes to reference images, replaces unreliable self-judgment with concrete measurements, and protects correct progress from regressive edits. Testing showed improvements across all models, with weaker agents seeing gains up to 62.7%, while the already strong GPT-6 Astra gained approximately two percentage points.

The researchers also investigated whether reconstructed scenes could serve as foundations for standard computer vision tasks. Since each scene exists as an executable program, object detection, segmentation, and depth estimation can be extracted directly from the code without additional training. Results were promising but not yet competitive with specialized models. Object detection performed best, reaching roughly half the performance of dedicated systems, while segmentation and depth estimation showed larger performance gaps.

The findings align with broader observations about GPT-6 Astra's capabilities in spatial reasoning, with some researchers suggesting the model may have been trained on substantial 3D data including Blender scenes. This has prompted 3D software companies to prepare for AI agent integration, with Unity already releasing official plugins for various AI coding systems.

The research represents a significant step toward AI systems that can understand and reconstruct three-dimensional environments from photographs, though substantial improvements in accuracy remain necessary before these approaches can match specialized computer vision models. The work particularly highlights the importance of objective measurement systems rather than relying on AI self-assessment for iterative improvement processes.