AI Dev Impact Lab JA
← Topics ranking · 2026-11
GITHUB TOPIC

#crawler

GitHub repositories that have self-applied the topic "crawler" — a creator-tagged metadata that surfaces how AI projects describe themselves.

4
tagged repos
3
top 4 stars
2
with tool sigs
4
shown

REPOS Repos for #crawler (top 4 by stars)

sftrkr/sitepulse

Rust CLI and MCP server for technical SEO, sitemap audits, broken link detection, and AI agent readiness.

Rust ★ 1 AI 70
VictorCreciun/crawlview

See what search engines and AI crawlers actually store for your pages — not what your browser shows you.

TypeScript ★ 1 AI 70 Solo live ↗
SolitudeZY/OpinionLens

舆见 OpinionLens · 社媒舆情监测与 AI 报告桌面端(Electron + FastAPI sidecar + MediaCrawler 运行时包装)。设置中心:邮件/企微/Bark 通知、多账号号池、OpenAI 兼容/DeepSeek/Anthropic 模型 API。

Python ★ 1 AI 50 1 sig
mark1russell7/crawler

polite-crawler: a polite web crawler for research and AI agents. Fetch once, keep it, read it many times. robots.txt, one request per site, full-text search, MCP server.

TypeScript ★ 0 AI 70 Solo 1 sig live ↗

RELATED Other topics · full topics ranking →

#claude-code

3,787

#ai-agents

3,201

#llm

2,090

#mcp

1,866

#claude

1,854

#python

1,678

#developer-tools

1,530

#codex

1,376

#ai

1,181

Aggregated by case-insensitive match against topics_json of each repo's latest content snapshot. methodology