Microsoft TRELLIS 2

Overview

Microsoft TRELLIS 2 is an open-source framework for native and compact structured latent 3D generation, released by Microsoft Research. Reaching 10,253 GitHub stars, TRELLIS 2 advances the state of 3D content generation by introducing a structured latent representation that enables high-fidelity, efficient 3D synthesis suitable for real-time applications.

TRELLIS 2 is directly relevant to the AI agent ecosystem as a content generation tool for agents. As AI agents increasingly generate and manipulate 3D assets for gaming, architecture, simulation, and AR/VR applications, frameworks like TRELLIS 2 provide the underlying generation capability that agents can invoke through APIs or MCP server integrations.

Features

• Structured latent representation: native compact latent space designed for 3D content, reducing memory and compute requirements
• High-fidelity generation: produces photorealistic 3D assets with fine geometric and texture detail
• Efficient inference: optimized for real-time or near-real-time generation, enabling agent-driven workflows
• Multimodal input support: accepts text, image, and sketch inputs for 3D generation
• Open-source: MIT-licensed, with full model weights and training code available
• Composable pipeline: modular architecture allows integration into existing agent pipelines

Installation

# Clone the repository
git clone https://github.com/microsoft/TRELLIS.2.git
cd TRELLIS.2

# Create virtual environment
python -m venv venv
source venv/bin/activate

# Install dependencies
pip install -r requirements.txt
pip install -e .

Quick Start

from trellis2 import TrellisPipeline

pipeline = TrellisPipeline.from_pretrained("microsoft/trellis2-base")

# Generate 3D from text
output = pipeline.generate(
    prompt="a modern coffee mug with latte art",
    steps=50,
    guidance_scale=7.5,
)

# Save generated mesh
output.mesh.save("output.obj")

Core Concepts

• Structured Latent Space (SLS): a compact representation of 3D content that preserves geometric structure while being efficiently encodable/decodable
• Diffusion-based generation: uses 3D-aware diffusion models trained on structured latents for high-quality generation
• Compact encoding: achieves 3D representation at a fraction of the memory cost of naive voxel or mesh representations
• Multimodal conditioning: supports text, image, and sketch inputs through unified conditioning adapters

Advanced Features

• Inpainting: edit specific regions of a generated 3D asset
• Style transfer: apply style from reference images to 3D generation
• Batch generation: efficient batch processing for large-scale content pipelines
• Agent integration: REST API and Python SDK designed for programmatic agent access

Examples

Scenario	Description
Game asset generation	Agent generates 3D props from text descriptions
AR/VR content	Agent creates 3D objects for augmented reality experiences
Architectural visualization	Agent generates 3D models from floor plans or descriptions
Product design	Agent iterates on 3D product prototypes via conversational prompts

Pros

• ✅ Open-source with MIT license
• ✅ Backed by Microsoft Research
• ✅ High-fidelity output suitable for production use
• ✅ Efficient structured latent representation
• ✅ 10K+ stars, active community
• ✅ Multimodal input support

Cons

• ❌ GPU required for practical generation speed
• ❌ Large model download (~5GB for base model)
• ❌ Limited to generation — no editing or physics simulation
• ❌ Early-stage framework, API subject to change

When to Use

TRELLIS 2 is ideal for AI agent workflows that require 3D content generation — game development, AR/VR experiences, product visualization, and simulation environments. Its compact representation and efficient inference make it suitable for agent-driven batch generation pipelines where agents invoke the framework programmatically to produce 3D assets from natural language descriptions.

Resources

• GitHub Repository
• Microsoft Research Blog
• Technical Paper