Skip to content
Agents tracked: 444 Releases (7d): 644 Agent status: 6 with issuesSince last limit reset: Claude 30d · Codex 3d Updated Oct 23, 2026
AgentGid
✦ Ask

Web research and scraping agent: product recipe

For an agent that searches the web, reads and scrapes pages (with a real browser when needed), runs code on what it finds and writes a sourced report. Every service below is from AgentGid's catalog, with the price and free tier from its official page; the first pick in each part ranks best on free tier, official MCP server, real use and price.

agent.md

What it costs

To start
$0 / month
every first pick has a free tier or is open source (limits apply)
If you outgrow the free tiers
about $218 / month
Crawl4AI $10, E2B $150, Supermemory $19, LangSmith $39; billed by usage: Exa, Browser Use Cloud
Model, per 1,000 research steps
about $1.6
Claude Haiku 5.5; $12 with Gemini 3.8 Flash; AgentGid estimate at ~12,000 input + 800 output tokens each

How it fits together

  1. Search the web for the question and pick the sources worth reading.
  2. Read pages with a scraping API; open a cloud browser only for pages that need clicks or logins.
  3. Run code in the sandbox for numbers and tables, not in your own server.
  4. Keep notes in memory between steps and write a report with a link for every claim.

Web search

Exa first pick
MCP
 Neural web search API with page contents, answers and deep-research agent runs for LLMs.
Free: $10 credits every month plus a $10 onboarding bonus, no payment method · Search from $4 per 1,000 requests (Lite $4, Auto $7, Deep $12); Contents $1 per 1,000 pages; Monitors $15 per… · in 100 public agent repos
npm install exa-js pip install exa-py agent.md
Tavily Web search, extract, crawl and research API that returns LLM-ready results for agents.
Free: 1,000 API credits/month, no credit card · paid from $30/mo · $0.008 per credit (pay as you go); monthly plans $0.0075-$0.005 per credit · in 377 public agent repos
npm install @tavily/core pip install tavily-python agent.md
SerpApi
MCP
 Scrapes Google and other search engines and returns structured JSON results.
Free: 250 searches/month · paid from $25/mo · Starter $25/month for 1,000 searches; Developer $75 for 5,000; Production $150 for 15,000; Big Data $275 for … · in 48 public agent repos
npm install serpapi pip install serpapi agent.md

Also fits: Parallel, Linkup, Perplexity API (Sonar / Search API), Brave Search API · all web search apis

Scraping and crawling

Crawl4AI first pick
MCP
 Open-source LLM-friendly crawler you can self-host, now also offered as a hosted cloud API.
Free: Self-host free (open source); Cloud Free: 1,000 credits/month (about 2,000 page scrapes), 10/min, 2 concurrent, no card · paid from $10/mo · Cloud: 1 credit = $0.001; page 0.5 credit, search 1, archive 0.25, real browser 1, hard site 2. Starter $10/m… · in 222 public agent repos
pip install crawl4ai agent.md
Firecrawl Scrapes, crawls, maps and searches websites, returning clean markdown or JSON for LLMs.
Free: 1,000 credits/month, no credit card (500 searches or 1,000 pages) · paid from $19/mo · 1 credit = 1 page scraped; Hobby extra credits 1,000 per $5 (Standard 2,000 per $5, Growth 2,500 per $5) · in 491 public agent repos
npm install @mendable/firecrawl-js pip install firecrawl-py agent.md
Apify
MCP
 Platform of ready-made scrapers (Actors) plus serverless runtime and proxies for web data.
Free: $5/month of platform usage, no credit card · paid from $19/mo · $0.20 per compute unit (1 GB RAM-hour) on Free/Starter, $0.16 Scale, $0.13 Business; residential proxy $8/GB · in 71 public agent repos
npm install apify-client pip install apify-client agent.md

Also fits: Spider, Cheerio, ScrapeGraphAI API, Bright Data (Unlocker API) · all scraping and crawling

Cloud browser

Browser Use Cloud first pick Hosted browsers and browser agents from the open-source Browser Use framework.
Free: $1 free credit once per person (eligible Google/GitHub/Microsoft signups) · Browsers $0.02 per hour; residential proxy $5/GB; agents = model token cost + 20% · in 258 public agent repos
npm install browser-use-sdk pip install browser-use agent.md
Browserbase
MCP
 Cloud headless browsers plus search, fetch and runtime APIs for browser agents (makers of Stagehand).
Free: 1 browser hour, 3 concurrent browsers, 1,000 Search and 1,000 Fetch calls · paid from $20/mo · Developer: 100 browser hours included then $0.12/browser hour; Startup ($99): 500 hours then $0.10/hour; prox… · in 97 public agent repos
npm install @browserbasehq/sdk pip install browserbase agent.md

Also fits: Puppeteer, Steel, Anchor Browser, Hyperbrowser · all browser automation (cloud)

Code sandbox

E2B first pick
MCP
 Secure cloud sandboxes (microVMs) where agents run code, start in under a second.
Free: Hobby: $100 one-time usage credit, sessions up to 1 hour, 20 concurrent sandboxes · paid from $150/mo · $0.000014/s per vCPU ($0.0504 per vCPU-hour) + $0.0000045/s per GiB RAM; default sandbox 2 vCPU / 4 GiB · in 85 public agent repos
npm install e2b pip install e2b agent.md

Also fits: Daytona, Vercel Sandbox, Modal Sandboxes, Riza · all code sandboxes

Memory

Supermemory first pick
MCP
 Memory and retrieval API that builds per-user memory graphs from text, files and connectors.
Free: $5 of credits/month · paid from $19/mo · Memory $5 per 1M SM tokens (plain text), $10 (rich); SuperRAG $1-$2 per 1M tokens; search $5 per 1M queries · in 21 public agent repos
npm install supermemory pip install supermemory agent.md

Also fits: Mem0, Letta, Zep · all memory for agents

Traces and evals

LangSmith first pick
MCP
 Tracing, evaluation and monitoring of LLM apps and agents, including prompt and dataset management.
Free: Developer plan: 1 seat, up to 5k base traces / mo, then pay-as-you-go · paid from $39/mo · Plus: $39 / seat per month incl. up to 10k base traces / mo; trace overage is pay-as-you-go (rate not listed … · in 678 public agent repos
npm install langsmith pip install langsmith agent.md

Also fits: LangWatch, W&B Weave, Ragas, Pydantic Logfire · all llm observability and evals

Model

Default: fast and cheapClaude Haiku 5.5 · $0.1 input / $0.5 output per 1M tokens · or GPT-6 Luna, MiMo V2.6 Flash
Harder questionsGemini 3.8 Flash · $0.75 input / $3.75 output per 1M tokens · or Kimi K2.7 Code, DeepSeek V4 Pro
Hardest workKimi K3 · $3 input / $15 output per 1M tokens · or Claude Opus 5.5, Claude Fable 5.1

The cheapest model of each price tier that can call tools, from model prices. Try the default first and move up only for the questions it gets wrong.

Or start from a ready platform

Browser UseOpen-source Python library and hosted cloud that let LLM agents control a real browser to complete web tasks, with a CLI, a cloud API for hosted agents and bro…
Free tier
Browser HarnessBrowser Harness | Self-healing harness that enables LLMs to complete any task.
Free tier
agent-browserNative Rust command-line tool that lets AI agents control Chrome through compact accessibility-tree snapshots and ref-based commands such as open, click and fi…
Free tier
StagehandOpen-source SDK from Browserbase for building browser agents that mix natural-language actions (act, extract, observe, agent) with code, available in TypeScrip…
Free tier · from $20/mo

Libraries

Crawl4AILLM-friendly web crawler and scraper for extracting data from websites.
pip install crawl4ai · Apache-2.0 · Gid Score 64
Browser UseSDK for AI agents to interact with websites via browser automation.
pip install browser-use · MIT · Gid Score 54
Playwright (Python)High-level Python API for automating browser testing and web scraping.
pip install playwright · Apache-2.0 · Gid Score 77
Firecrawl (Python)Web scraping and crawling SDK with search over PubMed and arXiv research papers.
pip install firecrawl-py · MIT · Gid Score 72
Beautiful SoupPython HTML/XML parsing and screen-scraping library for web data extraction.
pip install beautifulsoup4 · MIT · Gid Score 77
Pydantic AIAI agent framework built with Pydantic for structured, typed agent development.
pip install pydantic-ai · MIT · Gid Score 62
LangGraphFramework for building stateful, multi-actor applications with LLMs.
pip install langgraph · MIT · Gid Score 78

Agent Skills

anchor-browser-automation ComposioHQ/awesome-claude-skillsAutomate Anchor Browser tasks via Rube MCP (Composio). Always search tools first for current schemas.
npx skills add ComposioHQ/awesome-claude-skills --skill anchor-browser-automation
deep-research Imbad0202/academic-research-skillsUniversal deep research agent team. 13-agent pipeline for rigorous academic research on any topic. 8 modes: full research, quick brief, paper review, lit-review, fact-ch…
npx skills add Imbad0202/academic-research-skills --skill deep-research
browser-automation zhaoxuya520/reverse-skill统一自动化入口。覆盖浏览器自动化（Playwright）和 Windows 桌面应用自动化（OpenReverse）。 浏览器场景：打开网页、点击、填表、爬取、截图、自动化登录、渗透页面交互。 桌面场景：操作 IDA/x64dbg 等 GUI 工具、Windows UI Automation、视觉驱动交互、桌面应用网络抓包。 触发关键词… has scripts
npx skills add zhaoxuya520/reverse-skill --skill browser-automation
browser-automation alirezarezvani/claude-skills
npx skills add alirezarezvani/claude-skills --skill browser-automation
seo-firecrawl AgriciDaniel/claude-seoFull-site crawling, scraping, and site mapping via Firecrawl MCP. Use when user says "crawl site", "map site", "full crawl", "find all pages", "broken links", "site stru…
npx skills add AgriciDaniel/claude-seo --skill seo-firecrawl
anchor-browser-automation composio-community/awesome-codex-skillsAutomate Anchor Browser tasks via Rube MCP (Composio). Always search tools first for current schemas.
npx skills add composio-community/awesome-codex-skills --skill anchor-browser-automation

Read a skill before you install it: skills can run scripts.

Coding agent to build it

OpenAI Codex Claude Code OpenCode Pi Compare coding agents

Before you ship

  • Respect each site's terms and robots rules; do not use the browser to get around logins or captchas.
  • Keep API keys on the server, never in the browser or app; set spending limits where the provider offers them.
  • Read each service's terms for your use: data retention, commercial use, rate limits.
  • Log model calls (traces) and test with your own real inputs before launch.

The brief

# Product recipe: Web research and scraping agent (AgentGid)

For an agent that searches the web, reads and scrapes pages (with a real browser when needed), runs code on what it finds and writes a sourced report.

Parts were picked from AgentGid's catalog by rules: free tier, official MCP server, use in public projects built with AI agents, downloads, price. Prices are list prices from the official pages (see each card for the check date); confirm before you commit.

## What it costs
- Start: $0 a month on the free tiers of the first picks (their limits apply).
- If you outgrow the free tiers, the lowest paid plans of the first picks are Crawl4AI $10, E2B $150, Supermemory $19, LangSmith $39 (together about $218/month); usage beyond a plan's credits is billed on top.
- Billed by usage (grows with traffic): Exa, Browser Use Cloud.
- Model, default: fast and cheap: about $1.6 per 1,000 research steps with Claude Haiku 5.5 (AgentGid estimate at ~12,000 input + 800 output tokens each).
- Model, harder questions: about $12 per 1,000 research steps with Gemini 3.8 Flash (AgentGid estimate at ~12,000 input + 800 output tokens each).
- Prices of the picks below were checked on the official pages between 2026-10-10 and 2026-10-20.

## How it fits together
1. Search the web for the question and pick the sources worth reading.
2. Read pages with a scraping API; open a cloud browser only for pages that need clicks or logins.
3. Run code in the sandbox for numbers and tables, not in your own server.
4. Keep notes in memory between steps and write a report with a link for every claim.

## Web search
- Exa: free: $10 credits every month plus a $10 onboarding bonus, no payment method · usage: Search from $4 per 1,000 requests (Lite $4, Auto $7, Deep $12); Contents $1 per 1,000 pages; Monitors $15 per 1,000 requests; Agent $0.012–$1.00 per run · official MCP exa-mcp-server · npm exa-js · pip exa-py · card https://agentgid.com/apis/exa/agent.md
- Tavily: free: 1,000 API credits/month, no credit card · paid from $30/mo · usage: $0.008 per credit (pay as you go); monthly plans $0.0075-$0.005 per credit · official MCP tavily-mcp · npm @tavily/core · pip tavily-python · card https://agentgid.com/apis/tavily/agent.md · MCP server card https://agentgid.com/mcp/servers/io-github-tavily-ai-tavily-mcp/agent.md
- SerpApi: free: 250 searches/month · paid from $25/mo · usage: Starter $25/month for 1,000 searches; Developer $75 for 5,000; Production $150 for 15,000; Big Data $275 for 30,000 · official MCP https://mcp.serpapi.com/mcp · npm serpapi · pip serpapi · card https://agentgid.com/apis/serpapi/agent.md
- Also fits: Parallel, Linkup, Perplexity API (Sonar / Search API), Brave Search API (https://agentgid.com/apis/web-search/)

## Scraping and crawling
- Crawl4AI: free: Self-host free (open source); Cloud Free: 1,000 credits/month (about 2,000 page scrapes), 10/min, 2 concurrent, no card · paid from $10/mo · usage: Cloud: 1 credit = $0.001; page 0.5 credit, search 1, archive 0.25, real browser 1, hard site 2. Starter $10/mo 10,000 credits, Builder $99/mo 125,000, Scale $499/mo 750,000; extra credits $1/$0.79/$0.67 per 1,000 credits · official MCP https://api.crawl4ai.com/mcp · pip crawl4ai · card https://agentgid.com/apis/crawl4ai/agent.md
- Firecrawl: free: 1,000 credits/month, no credit card (500 searches or 1,000 pages) · paid from $19/mo · usage: 1 credit = 1 page scraped; Hobby extra credits 1,000 per $5 (Standard 2,000 per $5, Growth 2,500 per $5) · official MCP firecrawl-mcp · npm @mendable/firecrawl-js · pip firecrawl-py · card https://agentgid.com/apis/firecrawl/agent.md · MCP server card https://agentgid.com/mcp/servers/io-github-firecrawl-firecrawl-mcp-server/agent.md
- Apify: free: $5/month of platform usage, no credit card · paid from $19/mo · usage: $0.20 per compute unit (1 GB RAM-hour) on Free/Starter, $0.16 Scale, $0.13 Business; residential proxy $8/GB · official MCP @apify/actors-mcp-server · npm apify-client · pip apify-client · card https://agentgid.com/apis/apify/agent.md
- Also fits: Spider, Cheerio, ScrapeGraphAI API, Bright Data (Unlocker API) (https://agentgid.com/apis/scraping/)

## Cloud browser
- Browser Use Cloud: free: $1 free credit once per person (eligible Google/GitHub/Microsoft signups) · usage: Browsers $0.02 per hour; residential proxy $5/GB; agents = model token cost + 20% · official MCP https://api.browser-use.com/v3/mcp · npm browser-use-sdk · pip browser-use · card https://agentgid.com/apis/browser-use-cloud/agent.md · MCP server card https://agentgid.com/mcp/servers/com-browser-use-browser-use/agent.md
- Browserbase: free: 1 browser hour, 3 concurrent browsers, 1,000 Search and 1,000 Fetch calls · paid from $20/mo · usage: Developer: 100 browser hours included then $0.12/browser hour; Startup ($99): 500 hours then $0.10/hour; proxies $12/GB (Developer) · official MCP @browserbasehq/mcp · npm @browserbasehq/sdk · pip browserbase · card https://agentgid.com/apis/browserbase/agent.md
- Also fits: Puppeteer, Steel, Anchor Browser, Hyperbrowser (https://agentgid.com/apis/browser/)

## Code sandbox
- E2B: free: Hobby: $100 one-time usage credit, sessions up to 1 hour, 20 concurrent sandboxes · paid from $150/mo · usage: $0.000014/s per vCPU ($0.0504 per vCPU-hour) + $0.0000045/s per GiB RAM; default sandbox 2 vCPU / 4 GiB · official MCP @e2b/mcp-server · npm e2b · pip e2b · card https://agentgid.com/apis/e2b/agent.md
- Also fits: Daytona, Vercel Sandbox, Modal Sandboxes, Riza (https://agentgid.com/apis/sandbox/)

## Memory
- Supermemory: free: $5 of credits/month · paid from $19/mo · usage: Memory $5 per 1M SM tokens (plain text), $10 (rich); SuperRAG $1-$2 per 1M tokens; search $5 per 1M queries · official MCP https://mcp.supermemory.ai/mcp · npm supermemory · pip supermemory · card https://agentgid.com/apis/supermemory/agent.md
- Also fits: Mem0, Letta, Zep (https://agentgid.com/apis/memory/)

## Traces and evals
- LangSmith: free: Developer plan: 1 seat, up to 5k base traces / mo, then pay-as-you-go · paid from $39/mo · usage: Plus: $39 / seat per month incl. up to 10k base traces / mo; trace overage is pay-as-you-go (rate not listed on pricing page); 1 LSU = $1 (Deployment/Fleet/Sandbox usage) · official MCP https://docs.langchain.com/langsmith/langsmith-mcp-server · npm langsmith · pip langsmith · card https://agentgid.com/apis/langsmith/agent.md
- Also fits: LangWatch, W&B Weave, Ragas, Pydantic Logfire (https://agentgid.com/apis/observability/)

## Model (all of these can call tools; prices per 1M tokens)
- Default: fast and cheap: Claude Haiku 5.5 ($0.1 in / $0.5 out) · https://agentgid.com/models/claude-haiku-5-5/agent.md · or GPT-6 Luna, MiMo V2.6 Flash
- Harder questions: Gemini 3.8 Flash ($0.75 in / $3.75 out) · https://agentgid.com/models/gemini-3-8-flash/agent.md · or Kimi K2.7 Code, DeepSeek V4 Pro
- Hardest work: Kimi K3 ($3 in / $15 out) · https://agentgid.com/models/kimi-k3/agent.md · or Claude Opus 5.5, Claude Fable 5.1

## Or start from a ready platform
- Browser Use: free tier · https://agentgid.com/agents/browser-use/agent.md
- Browser Harness: free tier · https://agentgid.com/agents/browser-harness/agent.md
- agent-browser: free tier · https://agentgid.com/agents/agent-browser/agent.md
- Stagehand: free tier, from $20/mo · https://agentgid.com/agents/stagehand/agent.md

## Libraries
- `pip install crawl4ai`: Crawl4AI (Apache-2.0) · https://agentgid.com/tools/crawl4ai/agent.md
- `pip install browser-use`: Browser Use (MIT) · https://agentgid.com/tools/browser-use/agent.md
- `pip install playwright`: Playwright (Python) (Apache-2.0) · https://agentgid.com/tools/playwright/agent.md
- `pip install firecrawl-py`: Firecrawl (Python) (MIT) · https://agentgid.com/tools/firecrawl-py/agent.md
- `pip install beautifulsoup4`: Beautiful Soup (MIT) · https://agentgid.com/tools/beautifulsoup4/agent.md
- `pip install pydantic-ai`: Pydantic AI (MIT) · https://agentgid.com/tools/pydantic-ai/agent.md
- `pip install langgraph`: LangGraph (MIT) · https://agentgid.com/tools/langgraph/agent.md

## Agent Skills (read a skill before you install it: skills can run scripts)
- `npx skills add ComposioHQ/awesome-claude-skills --skill anchor-browser-automation`
- `npx skills add Imbad0202/academic-research-skills --skill deep-research`
- `npx skills add zhaoxuya520/reverse-skill --skill browser-automation`
- `npx skills add alirezarezvani/claude-skills --skill browser-automation`
- `npx skills add AgriciDaniel/claude-seo --skill seo-firecrawl`
- `npx skills add composio-community/awesome-codex-skills --skill anchor-browser-automation`

## Coding agent to build it
- OpenAI Codex: free tier, from $8/mo · https://agentgid.com/agents/openai-codex/agent.md
- Claude Code: from $20/mo · https://agentgid.com/agents/claude-code/agent.md
- OpenCode: free tier, from $10/mo · https://agentgid.com/agents/opencode/agent.md

## Before you ship
- Respect each site's terms and robots rules; do not use the browser to get around logins or captchas.
- Keep API keys on the server, never in the client; set spending limits where the provider offers them.
- Read each service's terms for your use (data retention, commercial use, rate limits).
- Log model calls (traces) and test with your own real inputs before launch.

Recipe page: https://agentgid.com/build/recipes/research-agent/ · all product recipes: https://agentgid.com/build/recipes/agent.md

Other recipes

AI assistant over your documents (RAG) Voice agent (phone or web) AI chatbot for a website or app AI support agent for your customers Telegram bot with AI Data analyst agent Solana trading or alert bot Video from a prompt Design recipes

Picked by rules from AgentGid's catalog, not paid placement. Prices are list prices from the official pages, checked between 2026-10-10 and 2026-10-20; check them before you commit.