Guidance

Overview

Guidance is a Microsoft research library that enables precise control over LLM generation through a unique programming model. It combines prompts, logic, and generation into a single unified interface, allowing developers to constrain outputs, enforce formats, and implement complex generation logic.

Unlike traditional prompting, Guidance lets you write programs that control the generation process at the token level, making it ideal for building reliable, structured LLM applications.

Features

• Unified Programming Model: Combine prompts, logic, and generation seamlessly
• Grammar-Based Constraints: Enforce output formats with grammars
• Stateful Generation: Capture and use generated variables
• Token-by-Token Control: Efficient generation with constraints
• Multi-Provider Support: OpenAI, Azure, local models
• Built-in Templating: Powerful template system for prompts
• Integration Ready: Works with existing codebases

Installation

pip install guidance

Quick Start

import guidance

# Initialize the model
model = guidance.models.OpenAI("gpt-4o")

# Define a guided generation
gen = guidance("""
    Given the following review, extract the sentiment and key points:

    Review: {{review_text}}

    Sentiment: {{#gen 'sentiment' choices=['positive', 'negative', 'neutral']}}
    Key Points: {{#gen 'key_points'}}
""")

# Run the generation
lm = model + gen(review_text="This product is amazing and works perfectly!")

print(lm["sentiment"])  # "positive"
print(lm["key_points"])  # "Works perfectly, amazing product"

Advanced Features

Grammar Constraints

import guidance

# Force JSON output
json_gen = guidance("""
    {{#gen 'json_output' grammar=json_grammar}}
""")

# Define JSON grammar
json_grammar = guidance.regex(r'\{.*\}')

lm = model + json_gen

Stateful Variables

lm = model + guidance("""
    {{#set 'name' value='Alice'}}
    Hello {{name}}!
    {{#gen 'response'}}
""")

# Use captured variable
print(lm["response"])

Conditional Logic

lm = model + guidance("""
    {{#if condition}}
        Do something
    {{else}}
        Do something else
    {{/if}}
    {{#gen 'output'}}
""")

Looping

lm = model + guidance("""
    {{#each items}}
        Item: {{this}}
    {{/each}}
    {{#gen 'summary'}}
""")

Use Cases

Structured Extraction

import guidance
from guidance import select

extractor = guidance("""
    Extract the following information from the text:

    Text: {{text}}

    Name: {{#gen 'name'}}
    Age: {{#gen 'age' regex=r'[0-9]+'}}
    City: {{#gen 'city'}}
""")

Constrained Generation

# Force specific output format
formal_response = guidance("""
    Respond in a formal tone:

    User: {{user_message}}

    Assistant: {{#gen 'response' temperature=0.7}}
""")

Multi-Step Reasoning

reasoning = guidance("""
    Let's think step by step.

    Question: {{question}}

    Step 1: {{#gen 'step1'}}
    Step 2: {{#gen 'step2'}}
    Step 3: {{#gen 'step3'}}

    Final Answer: {{#gen 'answer'}}
""")

Pros

• ✅ Precise control over LLM outputs
• ✅ Efficient generation with constraints
• ✅ Strong Microsoft research backing
• ✅ Good for structured generation tasks
• ✅ Open-source with permissive license
• ✅ Unique token-level control

Cons

• ❌ Niche use case (control-focused)
• ❌ Limited provider support
• ❌ Smaller community
• ❌ Documentation is sparse
• ❌ Steeper learning curve

When to Use

• Building applications requiring strict output formats
• Need deterministic, controlled generation
• Creating structured data extraction pipelines
• Research on LLM generation control
• When prompt engineering alone isn't enough

Use Cases

Use Case	Why Guidance
Structured Extraction	Force specific output formats with grammars
Constrained Generation	Control generation at token level
Multi-Step Reasoning	Implement complex generation logic
Research Applications	Study LLM generation control

Comparison with Alternatives

Feature	Guidance	Instructor	Outlines	LMQL
Paradigm	Programmatic	Pydantic-based	Grammar-based	Query-based
Token Control	✅ Yes	⚠️ Via retries	✅ Yes	⚠️ Limited
Type Safety	⚠️ Manual	✅ Pydantic	⚠️ Manual	⚠️ Manual
Multi-Provider	✅ Yes	✅ Yes	⚠️ Limited	⚠️ Limited
Learning Curve	High	Low	Medium	Medium
Best for	Research/control	Production extraction	Structured output	Query patterns

Best Practices

1. Define clear grammars — Use regex and JSON grammars for structure
2. Use stateful variables — Capture generated values with #set
3. Leverage templating — Use built-in template system for prompts
4. Test constraints — Verify grammars produce valid outputs
5. Start simple — Begin with basic generation before adding complexity

Troubleshooting

Issue	Solution
Grammar validation fails	Check regex patterns and JSON structure
Generation hangs	Verify constraints are satisfiable
Variables not captured	Use #set before referencing variables
Provider errors	Check model supports guided generation

Resources

• GitHub Repository
• Guidance Paper (arXiv)
• Examples