✓ Checked
Web Scraping: 10 GitHub Repos
A viral post lists 10 GitHub repos for web scraping. We checked each: license, language, releases, which ones AI-built projects use, why its Beautiful Soup link is not the official project, and Browserless's license.
On October 17, 2026 @DivyanshT91162 listed "10 GitHub repos that make web scraping seriously powerful": crawler frameworks, browser automation, article extractors, an MCP server and a hosted browser service.
We checked all ten on GitHub: license, language, latest release and how active each one is. Nine links go to the projects' own repositories. One does not: the Beautiful Soup link points to an archived personal mirror of an old version, 4.3.2, not to the project. And one project, Browserless, is not open source in the usual sense for commercial use.
The original post by @DivyanshT91162 on X (October 17, 2026)
Four kinds of tool on one list
The ten, checked
| Repo | Language | License | Last release | Note |
|---|---|---|---|---|
| Crawlee | JavaScript, TypeScript | Apache-2.0 | v4.0.0-rc.1, Oct 6, 2026 | A release candidate; a separate Python version exists |
| Scrapy | Python | BSD-3-Clause | 2.19.0, Sep 10, 2026 | Created in 2010 |
| Playwright | JavaScript, Python | Apache-2.0 | v1.64.1, Oct 14, 2026 | Built for testing and automation |
| Oxylabs Web API MCP | Python | MIT | none published | Created Aug 23, 2026; needs an Oxylabs Web API key |
| Colly | Go | Apache-2.0 | v2.2.0, Mar 27, 2025 | Commits continue; no release since March 2025 |
| Newspaper4k | Python | MIT | 0.9.6, Jul 19, 2026 | A fork that continues newspaper3k |
| trafilatura | Python and command line | Apache-2.0 | v2.3.1, Oct 6, 2026 | One main author |
| wention/BeautifulSoup4 | Python | not stated on GitHub | none | Archived mirror of 4.3.2; not the official project |
| MechanicalSoup | Python | MIT | v1.4.0, May 30, 2025 | No JavaScript |
| Browserless | Docker image | SSPL-1.0 or commercial | v2.57.1, Oct 16, 2026 | Commercial license for closed-source use |
Licenses, dates and releases from GitHub's API and each repository, read October 18, 2026.
The Beautiful Soup link is not the project
The post links to github.com/wention/BeautifulSoup4. GitHub describes that repository as a "git mirror for Beautiful Soup 4.3.2". The mirror was created in 2015, lists a single contributor, has been archived, and its last push was on November 8, 2022.
The official home of Beautiful Soup is crummy.com/software/BeautifulSoup, the site of its author Leonard Richardson. It says the current release is Beautiful Soup 4.14.3, from November 30, 2025, that the library is under the MIT license, and that development happens on Launchpad, not GitHub. Install it with pip install beautifulsoup4. Nothing is wrong with the library; the link is simply to an old copy, and anyone who clones it gets a version several years out of date.
Browserless is source-available, not free for business
Browserless's GitHub description says it is free for non-commercial uses. Its LICENSE file is precise: the code is dual-licensed under the MongoDB Server-Side Public License (SSPL-1.0) or the Browserless commercial license, and if you build commercial sites or applications, or use it in a closed-source continuous-integration system, you need to buy the commercial license. GitHub's API reports its license as NOASSERTION because it is not a standard open-source license. For a hobby project it is fine; for a company, budget for the license or use Playwright directly.
Stars
Oxylabs Web API MCP stands out at the bottom: 8 stars, created on August 23, 2026. It is a new, official client for Oxylabs' paid Web API rather than a scraping engine of its own; nothing in it works without an Oxylabs Web API key.
What AgentGid data shows
We scan 6,152 public GitHub projects set up for AI coding agents and record which libraries they depend on. Of the ten, these are on AgentGid Tools:
| Tool | Used in agent projects | Downloads, last 7 days |
|---|---|---|
| Playwright (JavaScript) | 1,049 | 76,192,279 (npm) |
| Beautiful Soup | 128 | 68,389,987 (PyPI) |
| Playwright (Python) | 69 | 26,246,389 (PyPI) |
| Trafilatura | 6 | 3,470,875 (PyPI) |
| Crawlee | 2 | 165,402 (npm) |
| Scrapy | 1 | 1,034,328 (PyPI) |
| Newspaper4k | 0 | 53,009 (PyPI) |
Colly (a Go module), MechanicalSoup, Browserless and the Oxylabs MCP server are not in our tools index. Two things stand out. Beautiful Soup and Playwright are what agent-built projects actually use; the big crawler frameworks barely appear, which suggests most of these projects fetch a few pages rather than crawl whole sites. And the old newspaper3k is still downloaded about 2.1 times as often as its maintained fork Newspaper4k, by our calculation (111,685 against 53,009 in the last 7 days), even though the README of Newspaper4k says newspaper3k has not been updated since September 2020. If your project still imports newspaper3k, the fork keeps the same API.
The post leaves out the tools built for feeding pages to language models. AgentGid tracks Crawl4AI and Firecrawl in agent tooling; Firecrawl and Apify, the company behind Crawlee, are also in our APIs for agents with prices and free tiers.
Respect robots.txt and site terms
Scraping tools do not decide what you are allowed to collect. A few plain rules:
- Read robots.txt and follow it. Scrapy's documentation says that with ROBOTSTXT_OBEY enabled Scrapy respects robots.txt, and that the setting is on in every project created with
scrapy startproject; leave it on. Colly lists robots.txt support among its features. - Read the site's terms. Many sites forbid automated collection in their terms of use, and personal data is protected by privacy law in many countries no matter how it was collected.
- Be gentle. Limit request rates, identify your crawler, cache what you already have, and prefer an official API or data export when the site offers one.
- Do not treat protections as obstacles to defeat. Some tools on the list advertise, as reported in their READMEs, browser fingerprint replication, proxy rotation or reading bot-protected pages. A login wall, a CAPTCHA or a block is the site owner telling you no. We do not cover ways around them, and you are responsible for how you use these tools.
What the post gets right
- Nine of the ten links are the official repositories, and all nine had commits in 2026; six of them also published a release this year.
- The one-line descriptions match the projects. Colly is a Go framework, trafilatura extracts main text and metadata, MechanicalSoup automates website interaction.
What it leaves out
- The Beautiful Soup link is an archived mirror of version 4.3.2, not the project.
- Browserless needs a paid license for closed-source commercial use.
- Oxylabs Web API MCP is a front end for a paid API, and is two months old.
- Crawlee's latest release is a release candidate of version 4; read its release notes before you pin it.
- MechanicalSoup does not run JavaScript, so it cannot read pages that build their content in the browser; its README says so.
- Nothing about robots.txt or terms of use. See above.
Where to start
- A few pages for an agent or a script: Beautiful Soup from PyPI, or trafilatura if you only want the article text.
- Pages that need JavaScript: Playwright, the most used of the ten in agent-built projects.
- Crawling a whole site you are allowed to crawl: Scrapy in Python, Crawlee in JavaScript or TypeScript, Colly in Go.
- News articles: Newspaper4k rather than the unmaintained newspaper3k.
- Hosted browsers at company scale: read Browserless's license first.
We did not run these tools; everything above comes from the repositories, their README and license files, the Beautiful Soup and Scrapy documentation sites, GitHub's API and AgentGid's tools index, read October 18, 2026.