Crawler Awesome

• bibliolater/bookhunter
  • MIT, Go
  • download tools for clawing the ebooks from internets.
• MontFerret/ferret
  • Apache-2.0, Go
  • Declarative
• PuerkitoBio/goquery
  • BSD-3, Golang
  • jQuery
• apify/impit
  • Apache-2.0, Rust
  • browser impersonation
  • npm
• NodeJS/Typescript
  • apify/crawlee
    • Apache-2.0, TS
    • web scraping, browser automation
  • Cheerio
  • Puppeteer
  • Playwright
• Web / Platform
  • firecrawl/web-agent
    • 面向结构化 Web research 的开源 Web data agent。
  • firecrawl/open-lovable
    • 把任意网站克隆/重建成现代 React 应用的工具。

Bots

• www.useragentstring.com/pages/Crawlerlist
• monperrus/crawler-user-agents/crawler-user-agents.json

Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/600.2.5 (KHTML, like Gecko) Version/8.0.2 Safari/600.2.5 (Amazonbot/0.1; +https://developer.amazon.com/support/amazonbot)