crawler

The API to search, scrape, and interact with the web at scale. 🔥

Scrapy, a fast high-level web crawling & scraping framework for Python.

DotnetSpider, a .NET standard web crawling library. It is lightweight, efficient and fast high-level web crawling & scraping framework

Web Scraping Framework

Async Python 3.6+ web scraping micro-framework based on asyncio

Fast, free, open-source technical SEO + GEO crawler: built for humans and agents.