← Back to tools

crawley

Unix-way web crawler that prints discovered links

Homepage
Development linuxmacoswindows Go MIT

Description

crawley is a Unix-way web crawler that crawls web pages and prints any link it can find. It follows the Unix philosophy of doing one thing well and outputting results to stdout.

AI Summary

A Unix-way web crawler that discovers and outputs all links found on web pages.

Capabilities

  • + Crawl web pages for links
  • + Unix-style stdout output
  • + Configurable crawl depth
  • + Filter and format output

Use When

  • → You need to discover links on web pages
  • → You want a simple, pipe-friendly web crawler

Avoid When

  • x You need full web scraping with content extraction
  • x You need JavaScript rendering
View AGENTS.md for crawley