Firecrawl MCP

Overview

Firecrawl is an API platform that enables search, scraping, and interaction with the web at scale. With over 149,000 GitHub stars, it has become the leading web data extraction platform for AI agents, providing LLM-ready output formats and autonomous data gathering capabilities. The platform covers 96% of the web including JavaScript-heavy pages, with a P95 latency of 3.4 seconds across millions of pages.

Firecrawl solves a critical challenge for AI agents: reliably extracting structured data from the web. Traditional scraping tools often fail on modern JavaScript-rendered pages, encounter rate limiting, or produce messy HTML that LLMs struggle to parse. Firecrawl addresses these issues by automatically handling rotating proxies, rate limits, and JavaScript rendering, while providing clean Markdown and structured JSON outputs optimized for AI consumption.

The platform offers an official MCP server, making it trivially easy to integrate web data extraction into AI coding agents like Claude Code, Cursor, and others. The Agent endpoint is particularly powerful, allowing agents to describe what data they need in natural language without specifying URLs.

Features

• Web Search: Search the web and get full page content from results
• Smart Scraping: Convert any URL to markdown, HTML, screenshots, or structured JSON
• Page Interaction: Scrape pages then interact using AI prompts or code (click, scroll, write, wait, press)
• Site Crawling: Automatically crawl entire websites and discover URLs
• Site Mapping: Discover all URLs on a website automatically
• Batch Scraping: Scrape multiple URLs in parallel
• Agent Mode: Autonomous data gathering based on natural language descriptions
• LLM-Ready Output: Clean Markdown and structured JSON optimized for AI consumption
• Media Parsing: Extract content from PDFs, DOCX files, and other formats
• High Coverage: 96% web coverage including JavaScript-rendered pages
• Performance: P95 latency of 3.4 seconds across millions of pages
• Automatic Handling: Proxies, orchestration, rate limits handled automatically

Installation

SDK Installation

Python:

pip install firecrawl-py

Node.js:

npm install firecrawl

Java: Add via Maven or Gradle from jitpack.io

Rust:

firecrawl = "2"

CLI Installation

npx -y firecrawl-cli@latest init --all --browser

Works with Claude Code, Antigravity, OpenCode, and other agent platforms.

MCP Server Configuration

Quick Setup

1. Sign up at firecrawl.dev to obtain an API key
2. Add Firecrawl MCP server to your MCP configuration

{
  "mcpServers": {
    "firecrawl-mcp": {
      "command": "npx",
      "args": ["-y", "firecrawl-mcp"],
      "env": {
        "FIRECRAWL_API_KEY": "fc-YOUR_API_KEY"
      }
    }
  }
}

Available Tools

Tool	Description
firecrawl_scrape	Scrape a single URL to markdown, HTML, screenshot, or JSON
firecrawl_search	Search the web and get full page content from results
firecrawl_crawl	Crawl an entire website and extract all URLs
firecrawl_map	Discover all URLs on a website
firecrawl_batch	Scrape multiple URLs in parallel
firecrawl_agent	Autonomous data gathering with natural language prompts
firecrawl_extract	Extract structured data using schemas
firecrawl_screenshot	Capture screenshots of web pages

Usage Examples

Basic Scrape Operation

from firecrawl import Firecrawl

app = Firecrawl(api_key="fc-YOUR_API_KEY")

# Scrape a URL to markdown
result = app.scrape(
    url="https://example.com",
    format="markdown",
    screenshot=True
)
print(result["markdown"])

Agent Mode - Natural Language Data Gathering

# Find founders of Stripe without specifying URLs
result = app.agent(
    prompt="Find the founders of Stripe"
)
print(result)

Structured Output with Schema

from pydantic import BaseModel, Field
from typing import List

class Founder(BaseModel):
    name: str
    role: str
    background: str

class FoundersSchema(BaseModel):
    founders: List[Founder] = Field(description="List of founders")

result = app.agent(
    prompt="Find the founders of Firecrawl",
    schema=FoundersSchema
)
print(result)

Web Search with Full Content

# Search and get full page content from results
results = app.search(
    query="AI agent frameworks 2026",
    limit=5,
    format="markdown"
)
for result in results:
    print(result["url"])
    print(result["markdown"])

Page Interaction

# Scrape page then interact with it
result = app.scrape(
    url="https://example.com/products",
    actions=[
        {"type": "click", "selector": "button.load-more"},
        {"type": "wait", "milliseconds": 1000},
        {"type": "scroll", "direction": "down"},
        {"type": "click", "selector": "a.next-page"}
    ]
)

Site Crawling

# Crawl an entire website
job_id = app.crawl(
    url="https://docs.example.com",
    filters={
        "include_paths": ["/docs/*"],
        "exclude_paths": ["/blog/*"]
    },
    limit=100
)

# Get results
results = app.get_crawl_status(job_id)

Agent Usage Examples

Research Workflow

User: "Find all Python agent frameworks with over 10k stars"
Agent: Uses firecrawl_search to find relevant repositories
Agent: Uses firecrawl_scrape to extract details from each
Agent: "Found 12 frameworks: LangGraph (24k), CrewAI (18k), AutoGen (22k)..."

Competitive Analysis

User: "Analyze pricing for AI coding tools"
Agent: Uses firecrawl_agent to gather pricing information
Agent: "Claude Code: $20/mo, Cursor: $20/mo, Copilot: $10/mo..."

Documentation Extraction

User: "Get the latest LangGraph API documentation"
Agent: Uses firecrawl_scrape with markdown format
Agent: "Extracted complete API reference with 47 endpoints"

Advanced Features

Custom Headers and Authentication

result = app.scrape(
    url="https://private-api.example.com/docs",
    headers={
        "Authorization": "Bearer token123",
        "X-API-Key": "secret"
    }
)

Selective Extraction

# Extract only specific content
result = app.extract(
    url="https://example.com/article",
    schema={
        "title": "article title",
        "author": "author name",
        "date": "publication date",
        "content": "main content"
    }
)

Batch Operations

# Scrape multiple URLs in parallel
urls = [
    "https://docs.pydantic.dev",
    "https://docs.langchain.com",
    "https://docs.crewai.com"
]

results = app.batch_scrape(
    urls=urls,
    format="markdown"
)

Claude Desktop Setup

1. Sign up at firecrawl.dev and get API key
2. Add Firecrawl MCP server to Claude Desktop configuration
3. Restart Claude Desktop
4. Web scraping tools will be available in conversations

Cursor Setup

1. Install Firecrawl MCP server
2. Add to Cursor MCP settings with API key
3. Restart Cursor
4. Use scraping tools in agent conversations

Pros

• ✅ Highest web coverage (96%) including JavaScript pages
• ✅ Excellent performance (3.4s P95 latency)
• ✅ LLM-optimized output formats (Markdown, JSON)
• ✅ Abstracts away proxies, rate limits, rendering complexity
• ✅ Powerful Agent mode for natural language data gathering
• ✅ Multiple SDK languages (Python, Node.js, Java, Rust, Elixir)
• ✅ Official MCP server for easy AI integration
• ✅ Open source with 149,000+ stars
• ✅ Handles PDFs, DOCX, and other media formats

Cons

• ❌ Requires API key registration for hosted service
• ❌ Open source version lacks some cloud features
• ❌ Self-hosting requires additional setup
• ❌ Must respect website policies and robots.txt
• ❌ Advanced Agent models incur additional costs
• ❌ Rate limits on free tier

When to Use

Firecrawl MCP is ideal for:

• AI Research Agents: Autonomous web data gathering
• Competitive Intelligence: Monitoring competitor websites and pricing
• Content Analysis: Extracting structured data from web pages
• Documentation Mining: Gathering API references and documentation
• Market Research: Analyzing trends and industry data
• Lead Generation: Extracting contact information from websites
• SEO Analysis: Crawling and analyzing website content
• Agent Workflows: Any agent that needs real-time web data

Resources

• GitHub Repository
• Firecrawl Platform
• Documentation
• Pricing