# Product recipe: Web research and scraping agent (AgentGid)

For an agent that searches the web, reads and scrapes pages (with a real browser when needed), runs code on what it finds and writes a sourced report.

Parts were picked from AgentGid's catalog by rules: free tier, official MCP server, use in public projects built with AI agents, downloads, price. Prices are list prices from the official pages (see each card for the check date); confirm before you commit.

## What it costs
- Start: $0 a month on the free tiers of the first picks (their limits apply).
- If you outgrow the free tiers, the lowest paid plans of the first picks are Crawl4AI $10, E2B $150, Supermemory $19, LangSmith $39 (together about $218/month); usage beyond a plan's credits is billed on top.
- Billed by usage (grows with traffic): Exa, Browser Use Cloud.
- Model, default: fast and cheap: about $1.6 per 1,000 research steps with Claude Haiku 5.5 (AgentGid estimate at ~12,000 input + 800 output tokens each).
- Model, harder questions: about $12 per 1,000 research steps with Gemini 3.8 Flash (AgentGid estimate at ~12,000 input + 800 output tokens each).
- Prices of the picks below were checked on the official pages between 2026-10-10 and 2026-10-20.

## How it fits together
1. Search the web for the question and pick the sources worth reading.
2. Read pages with a scraping API; open a cloud browser only for pages that need clicks or logins.
3. Run code in the sandbox for numbers and tables, not in your own server.
4. Keep notes in memory between steps and write a report with a link for every claim.

## Web search
- Exa: free: $10 credits every month plus a $10 onboarding bonus, no payment method · usage: Search from $4 per 1,000 requests (Lite $4, Auto $7, Deep $12); Contents $1 per 1,000 pages; Monitors $15 per 1,000 requests; Agent $0.012–$1.00 per run · official MCP exa-mcp-server · npm exa-js · pip exa-py · card https://agentgid.com/apis/exa/agent.md
- Tavily: free: 1,000 API credits/month, no credit card · paid from $30/mo · usage: $0.008 per credit (pay as you go); monthly plans $0.0075-$0.005 per credit · official MCP tavily-mcp · npm @tavily/core · pip tavily-python · card https://agentgid.com/apis/tavily/agent.md · MCP server card https://agentgid.com/mcp/servers/io-github-tavily-ai-tavily-mcp/agent.md
- SerpApi: free: 250 searches/month · paid from $25/mo · usage: Starter $25/month for 1,000 searches; Developer $75 for 5,000; Production $150 for 15,000; Big Data $275 for 30,000 · official MCP https://mcp.serpapi.com/mcp · npm serpapi · pip serpapi · card https://agentgid.com/apis/serpapi/agent.md
- Also fits: Parallel, Linkup, Perplexity API (Sonar / Search API), Brave Search API (https://agentgid.com/apis/web-search/)

## Scraping and crawling
- Crawl4AI: free: Self-host free (open source); Cloud Free: 1,000 credits/month (about 2,000 page scrapes), 10/min, 2 concurrent, no card · paid from $10/mo · usage: Cloud: 1 credit = $0.001; page 0.5 credit, search 1, archive 0.25, real browser 1, hard site 2. Starter $10/mo 10,000 credits, Builder $99/mo 125,000, Scale $499/mo 750,000; extra credits $1/$0.79/$0.67 per 1,000 credits · official MCP https://api.crawl4ai.com/mcp · pip crawl4ai · card https://agentgid.com/apis/crawl4ai/agent.md
- Firecrawl: free: 1,000 credits/month, no credit card (500 searches or 1,000 pages) · paid from $19/mo · usage: 1 credit = 1 page scraped; Hobby extra credits 1,000 per $5 (Standard 2,000 per $5, Growth 2,500 per $5) · official MCP firecrawl-mcp · npm @mendable/firecrawl-js · pip firecrawl-py · card https://agentgid.com/apis/firecrawl/agent.md · MCP server card https://agentgid.com/mcp/servers/io-github-firecrawl-firecrawl-mcp-server/agent.md
- Apify: free: $5/month of platform usage, no credit card · paid from $19/mo · usage: $0.20 per compute unit (1 GB RAM-hour) on Free/Starter, $0.16 Scale, $0.13 Business; residential proxy $8/GB · official MCP @apify/actors-mcp-server · npm apify-client · pip apify-client · card https://agentgid.com/apis/apify/agent.md
- Also fits: Spider, Cheerio, ScrapeGraphAI API, Bright Data (Unlocker API) (https://agentgid.com/apis/scraping/)

## Cloud browser
- Browser Use Cloud: free: $1 free credit once per person (eligible Google/GitHub/Microsoft signups) · usage: Browsers $0.02 per hour; residential proxy $5/GB; agents = model token cost + 20% · official MCP https://api.browser-use.com/v3/mcp · npm browser-use-sdk · pip browser-use · card https://agentgid.com/apis/browser-use-cloud/agent.md · MCP server card https://agentgid.com/mcp/servers/com-browser-use-browser-use/agent.md
- Browserbase: free: 1 browser hour, 3 concurrent browsers, 1,000 Search and 1,000 Fetch calls · paid from $20/mo · usage: Developer: 100 browser hours included then $0.12/browser hour; Startup ($99): 500 hours then $0.10/hour; proxies $12/GB (Developer) · official MCP @browserbasehq/mcp · npm @browserbasehq/sdk · pip browserbase · card https://agentgid.com/apis/browserbase/agent.md
- Also fits: Puppeteer, Steel, Anchor Browser, Hyperbrowser (https://agentgid.com/apis/browser/)

## Code sandbox
- E2B: free: Hobby: $100 one-time usage credit, sessions up to 1 hour, 20 concurrent sandboxes · paid from $150/mo · usage: $0.000014/s per vCPU ($0.0504 per vCPU-hour) + $0.0000045/s per GiB RAM; default sandbox 2 vCPU / 4 GiB · official MCP @e2b/mcp-server · npm e2b · pip e2b · card https://agentgid.com/apis/e2b/agent.md
- Also fits: Daytona, Vercel Sandbox, Modal Sandboxes, Riza (https://agentgid.com/apis/sandbox/)

## Memory
- Supermemory: free: $5 of credits/month · paid from $19/mo · usage: Memory $5 per 1M SM tokens (plain text), $10 (rich); SuperRAG $1-$2 per 1M tokens; search $5 per 1M queries · official MCP https://mcp.supermemory.ai/mcp · npm supermemory · pip supermemory · card https://agentgid.com/apis/supermemory/agent.md
- Also fits: Mem0, Letta, Zep (https://agentgid.com/apis/memory/)

## Traces and evals
- LangSmith: free: Developer plan: 1 seat, up to 5k base traces / mo, then pay-as-you-go · paid from $39/mo · usage: Plus: $39 / seat per month incl. up to 10k base traces / mo; trace overage is pay-as-you-go (rate not listed on pricing page); 1 LSU = $1 (Deployment/Fleet/Sandbox usage) · official MCP https://docs.langchain.com/langsmith/langsmith-mcp-server · npm langsmith · pip langsmith · card https://agentgid.com/apis/langsmith/agent.md
- Also fits: LangWatch, W&B Weave, Ragas, Pydantic Logfire (https://agentgid.com/apis/observability/)

## Model (all of these can call tools; prices per 1M tokens)
- Default: fast and cheap: Claude Haiku 5.5 ($0.1 in / $0.5 out) · https://agentgid.com/models/claude-haiku-5-5/agent.md · or GPT-6 Luna, MiMo V2.6 Flash
- Harder questions: Gemini 3.8 Flash ($0.75 in / $3.75 out) · https://agentgid.com/models/gemini-3-8-flash/agent.md · or Kimi K2.7 Code, DeepSeek V4 Pro
- Hardest work: Kimi K3 ($3 in / $15 out) · https://agentgid.com/models/kimi-k3/agent.md · or Claude Opus 5.5, Claude Fable 5.1

## Or start from a ready platform
- Browser Use: free tier · https://agentgid.com/agents/browser-use/agent.md
- Browser Harness: free tier · https://agentgid.com/agents/browser-harness/agent.md
- agent-browser: free tier · https://agentgid.com/agents/agent-browser/agent.md
- Stagehand: free tier, from $20/mo · https://agentgid.com/agents/stagehand/agent.md

## Libraries
- `pip install crawl4ai`: Crawl4AI (Apache-2.0) · https://agentgid.com/tools/crawl4ai/agent.md
- `pip install browser-use`: Browser Use (MIT) · https://agentgid.com/tools/browser-use/agent.md
- `pip install playwright`: Playwright (Python) (Apache-2.0) · https://agentgid.com/tools/playwright/agent.md
- `pip install firecrawl-py`: Firecrawl (Python) (MIT) · https://agentgid.com/tools/firecrawl-py/agent.md
- `pip install beautifulsoup4`: Beautiful Soup (MIT) · https://agentgid.com/tools/beautifulsoup4/agent.md
- `pip install pydantic-ai`: Pydantic AI (MIT) · https://agentgid.com/tools/pydantic-ai/agent.md
- `pip install langgraph`: LangGraph (MIT) · https://agentgid.com/tools/langgraph/agent.md

## Agent Skills (read a skill before you install it: skills can run scripts)
- `npx skills add ComposioHQ/awesome-claude-skills --skill anchor-browser-automation`
- `npx skills add Imbad0202/academic-research-skills --skill deep-research`
- `npx skills add zhaoxuya520/reverse-skill --skill browser-automation`
- `npx skills add alirezarezvani/claude-skills --skill browser-automation`
- `npx skills add AgriciDaniel/claude-seo --skill seo-firecrawl`
- `npx skills add composio-community/awesome-codex-skills --skill anchor-browser-automation`

## Coding agent to build it
- OpenAI Codex: free tier, from $8/mo · https://agentgid.com/agents/openai-codex/agent.md
- Claude Code: from $20/mo · https://agentgid.com/agents/claude-code/agent.md
- OpenCode: free tier, from $10/mo · https://agentgid.com/agents/opencode/agent.md

## Before you ship
- Respect each site's terms and robots rules; do not use the browser to get around logins or captchas.
- Keep API keys on the server, never in the client; set spending limits where the provider offers them.
- Read each service's terms for your use (data retention, commercial use, rate limits).
- Log model calls (traces) and test with your own real inputs before launch.

Recipe page: https://agentgid.com/build/recipes/research-agent/ · all product recipes: https://agentgid.com/build/recipes/agent.md