Crawl4AI
LLM-friendly web crawler and scraper for extracting data from websites.
Scraping, browsers and parsing PyPI Apache-2.0 HTML parsing and extraction Browser automation
Website GitHub PyPI Its AgentGid profile Hosted service: prices
How Crawl4AI is used
Crawl4AI is a dependency in 9 of the 6,211 public GitHub projects set up for AI coding agents that we scanned, too few to describe it from them alone. Our scan also samples projects by dependency (Solana 2,717, Ethereum and EVM 1,533, 3D and games 2,862, agent frameworks and SDKs 11,237); counting those, it appears in 222 projects, and what it is used with below comes from all of them. The most used alternative in our scan is Beautiful Soup (134 projects). Projects that use it most often also use python-dotenv (64.0%), Pydantic (53.6%), Requests (53.2%).
Crawl4AI popularity over time
Gid Score 64: downloads 53, use in agent projects 62, GitHub stars 86, downloads trend 58 (each 0-100). How it is calculated.
Often used with Crawl4AI
| python-dotenv | 64.0% of Crawl4AI projects · 7.0× more than average |
|---|---|
| Pydantic | 53.6% of Crawl4AI projects · 6.2× more than average |
| Requests | 53.2% of Crawl4AI projects · 7.8× more than average |
| Beautiful Soup | 48.2% of Crawl4AI projects · 22.3× more than average |
| Playwright (Python) | 46.8% of Crawl4AI projects · 39.3× more than average |
| OpenAI Python Library | 46.4% of Crawl4AI projects · 11.8× more than average |
| FastAPI | 41.0% of Crawl4AI projects · 5.5× more than average |
| Uvicorn | 39.6% of Crawl4AI projects · 5.2× more than average |
From all 222 projects in our scan that use Crawl4AI, including the dependency samples.
By coding agent
| Cursor | 0.3% of projects set up for it |
|---|
Share of public repos configured for each agent that list Crawl4AI as a dependency.
How Crawl4AI is funded
GitHub Sponsors
| @unclecode | 11 current sponsors |
|---|
GitHub shows how many people and companies sponsor an account, not how much they pay. Sponsors of a person back all of their work.
Ways to fund Crawl4AI
As the maintainers list them in FUNDING.yml on GitHub. Checked Oct 20, 2026.
Install
| Package | pip install crawl4ai |
|---|---|
| Main stacks | Python (219), Next.js (2), .NET (1) |
| Example projects | camel-ai/owl, sentient-agi/OpenDeepSearch, zstar1003/ragflow-plus, polyuiislab/infiAgent, ustc-ai4science/Science-Star |
Alternatives to Crawl4AI
Tools that do the same job, so a project picks one or the other. Add-ons and libraries used alongside Crawl4AI are not listed here; they are in "Often used with".
| Tool | Gid Score | Used in agent projects | Downloads / wk | GitHub stars | License |
|---|---|---|---|---|---|
| Beautiful Soup pypi Python HTML/XML parsing and screen-scraping library for web data extraction. |
77 | 134 (8.3%) | 70.4M | — | MIT |
| Playwright (Python) pypi High-level Python API for automating browser testing and web scraping. |
77 | 74 (4.6%) | 28.4M | 15.1K | Apache-2.0 |
| Selenium pypi Official Python bindings for browser automation and web scraping. |
62 | 27 (1.7%) | 7.3M | — | Apache-2.0 |
| html2text pypi Convert HTML to Markdown-structured text for parsing and transforming web content. |
54 | 9 (0.6%) | 2.9M | 2,173 | GPL-3.0-or-later |
More in Scraping, browsers and parsing: Puppeteer lxml Playwright Core Cheerio fast-xml-parser jsonc-parser Mozilla Readability linkedom