Skip to content
Agents tracked: 492 Releases (7d): 545 Agent status: 1 with issuesSince last limit reset: Claude 9d · Codex 7d For AI agents: Markdown Updated Nov 20, 2026

Scrapy

Framework for crawling websites and extracting structured data at scale.

Scraping, browsers and parsingPYPI

GitHubWebsite

1. License

License from PyPI license expression.

github.com/scrapy/scrapy

From the package registry and GitHub, checked daily. Tap a verdict to see its source.How we checkedReport an error

What Scrapy does

Framework for crawling websites and extracting structured data at scale.

Scrapy popularity

59 / 100
Gid Score, #1004 overall
769K
downloads a week, pypi
65.4K
GitHub stars
1
agent projects use it
Scrapy: downloads per weekPyPI, scrapy 779,812
Scrapy: GitHub starsscrapy/scrapy 65,440

Gid Score measures public popularity and activity, not quality. How the Gid Score is calculated

How Scrapy is used

  • Scrapy is a dependency in 1 of the 6,254 public GitHub projects set up for AI coding agents that we scanned, too few to describe it from them alone. Our scan also samples projects by dependency (Solana 3,063, Ethereum and EVM 1,777, 3D and games 3,120, agent frameworks and SDKs 13,629); counting those, it appears in 40 projects, and what it is used with below comes from all of them.
  • The most used alternative in our scan is Beautiful Soup (135 projects).
  • Projects that use it most often also use pandas (82.5%), Requests (82.5%), Beautiful Soup (77.5%).

Often used with

In 82.5% of its projects
In 82.5% of its projects
In 77.5% of its projects
In 72.5% of its projects
In 70.0% of its projects
In 67.5% of its projects
In 65.0% of its projects
In 65.0% of its projects

More about Scrapy

More in Scraping, browsers and parsing

Alternatives

Scrapy This page

BSD-3-ClauseGid 59769K/wk

Beautiful Soup

Python HTML/XML parsing and screen-scraping library for web data extraction.

MITGid 7873.9M/wk

lxml

High-performance XML and HTML processing library combining libxml2/libxslt with Python.

BSD-3-ClauseGid 7398M/wk

Playwright (Python)

High-level Python API for automating browser testing and web scraping.

Apache-2.0Gid 7733.4M/wk

Selenium

Official Python bindings for browser automation and web scraping.

Apache-2.0Gid 626.4M/wk

Compare Scrapy with another tool

Get started

Install it.

pip install scrapy

No step-by-step start for Scrapy yet: we have not written its hand-checked card.

AgentGid articles about Scrapy

All guides