Promptfoo

Overview

Promptfoo is an open-source framework for testing and evaluating LLM prompts, models, and outputs. It provides a comprehensive suite of evaluation metrics, supports multiple LLM providers, and integrates with CI/CD pipelines.

Use Promptfoo to compare prompt variations, test model outputs, ensure your LLM applications behave correctly before deployment, and catch regressions early.

Features

• Multi-Provider Testing: Test across OpenAI, Anthropic, Google, and more
• Built-in Evaluation Metrics: Similarity, hallucination, toxicity, and more
• Prompt Versioning: Track and compare prompt changes over time
• CI/CD Integration: GitHub Actions, GitLab CI, and more
• Web UI: Visualize results and compare runs
• Custom Evaluators: Define your own evaluation logic
• Batch Testing: Test against large datasets

Installation

npm install -g promptfoo

Or for local development:

npm install promptfoo

Quick Start

Basic Test

# promptfoo.config.yaml
prompts:
  - "Summarize this: {{text}}"
  - "Briefly summarize: {{text}}"

tests:
  - vars:
      text: "The quick brown fox jumps over the lazy dog."
    expected: "A fox jumps over a dog."
  - vars:
      text: "Artificial intelligence is transforming industries."
    expected: "AI is changing various sectors."

promptfoo run

Compare Models

# promptfoo.config.yaml
providers:
  - openai:gpt-4o
  - anthropic:claude-3-5-sonnet-20241022
  - google:gemini-1.5-pro

prompts:
  - "Translate to French: {{text}}"

tests:
  - vars:
      text: "Hello, how are you?"
  - vars:
      text: "The meeting is at 3pm tomorrow."

Evaluation Metrics

Built-in Metrics

tests:
  - vars:
      text: "..."
    assert:
      - type: contains
        value: "expected phrase"
      - type: icontains
        value: "case insensitive"
      - type: regex
        value: "^\\d{3}-\\d{3}-\\d{4}$"
      - type: javascript
        value: "output.length < 100"
      - type: similar
        value: "expected output"
        threshold: 0.8
      - type: llm-rubric
        value: "Response should be helpful and accurate"
      - type: answer-relevance
        threshold: 0.7
      - type: context-recall
        threshold: 0.8

Custom Evaluators

// my-evaluator.js
module.exports = {
  name: 'custom-metric',
  validate(output, context) {
    // Your custom logic
    return {
      pass: output.includes('expected'),
      score: 0.9,
      reason: 'Output contains expected content'
    };
  }
};

tests:
  - assert:
      - type: custom-metric

CI/CD Integration

GitHub Actions

# .github/workflows/promptfoo.yml
name: Promptfoo Tests

on: [push, pull_request]

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: '20'
      - run: npm install -g promptfoo
      - run: promptfoo run
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}

Web UI

promptfoo view

Opens a local web interface at http://localhost:15500 where you can:

• View test results
• Compare different prompts
• Analyze model performance
• Export reports

Dataset Testing

tests:
  - file://tests.json

// tests.json
[
  {
    "vars": { "text": "..." },
    "expected": "..."
  },
  {
    "vars": { "text": "..." },
    "expected": "..."
  }
]

Pros

• ✅ Open-source and free to use
• ✅ Excellent for prompt iteration and testing
• ✅ CI/CD integration for automated testing
• ✅ Supports many LLM providers
• ✅ Good documentation and examples
• ✅ Visual comparison interface

Cons

• ❌ Not an agent framework (testing tool only)
• ❌ Evaluation metrics may need customization
• ❌ Web UI requires separate setup
• ❌ Smaller ecosystem than LangSmith

When to Use

• Iterating on prompts and comparing variations
• Testing LLM outputs before deployment
• Building CI/CD pipelines for LLM applications
• Evaluating model performance across providers
• Catching regressions in prompt changes

Use Cases

Use Case	Why Promptfoo
Prompt Testing	Compare prompt variations systematically
Model Evaluation	Test outputs across multiple LLM providers
CI/CD Integration	Automated testing in deployment pipelines
Regression Detection	Catch prompt changes that break behavior

Comparison with Alternatives

Feature	Promptfoo	LangSmith	Arize Phoenix	DeepEval
Open Source	✅ Yes	❌ No	✅ Yes	✅ Yes
Multi-Provider	✅ 100+	⚠️ Limited	⚠️ Limited	✅ Yes
CI/CD Native	✅ Yes	⚠️ Limited	❌ No	✅ Yes
Web UI	✅ Yes	✅ Yes	✅ Yes	❌ No
Custom Evaluators	✅ Yes	⚠️ Limited	⚠️ Limited	✅ Yes
Learning Curve	Low	Medium	Medium	Medium
Best for	Prompt testing	LangChain deep dive	Open-source tracing	Python evals

Best Practices

1. Define test datasets early — Create comprehensive test cases
2. Use multiple evaluators — Combine similarity, hallucination, toxicity checks
3. Integrate with CI/CD — Run tests on every prompt change
4. Compare models systematically — Test same prompts across providers
5. Track prompt versions — Use versioning to compare iterations

Troubleshooting

Issue	Solution
Tests not running	Check config.yaml syntax and file paths
Custom evaluator fails	Validate JavaScript function returns correct format
Results not comparing	Ensure consistent test variables across runs
CI/CD fails	Verify API keys are set in environment variables

Resources

• Official Documentation
• GitHub Repository
• Examples Gallery
• Discord Community