mark1russell7/crawler

polite-crawler: a polite web crawler for research and AI agents. Fetch once, keep it, read it many times. robots.txt, one request per site, full-text search, MCP server.

GitHub

Type

Research

Industry

Not stated

Deployment

Not stated

Stack

MCPTypeScript