#crawler
GitHub repositories that have self-applied the topic "crawler" — a creator-tagged metadata that surfaces how AI projects describe themselves.
REPOS Repos for #crawler (top 4 by stars)
Rust CLI and MCP server for technical SEO, sitemap audits, broken link detection, and AI agent readiness.
VictorCreciun/crawlviewSee what search engines and AI crawlers actually store for your pages — not what your browser shows you.
SolitudeZY/OpinionLens舆见 OpinionLens · 社媒舆情监测与 AI 报告桌面端(Electron + FastAPI sidecar + MediaCrawler 运行时包装)。设置中心:邮件/企微/Bark 通知、多账号号池、OpenAI 兼容/DeepSeek/Anthropic 模型 API。
mark1russell7/crawlerpolite-crawler: a polite web crawler for research and AI agents. Fetch once, keep it, read it many times. robots.txt, one request per site, full-text search, MCP server.
RELATED Other topics · full topics ranking →
#claude-code
3,787#ai-agents
3,201#llm
2,090#mcp
1,866#claude
1,854#python
1,678#developer-tools
1,530#codex
1,376#ai
1,181Aggregated by case-insensitive match against topics_json of each repo's latest content snapshot. methodology