What we fetch and from where
aidev-index relies exclusively on the public APIs and datasets below. The core architecture: GH Archive (BigQuery) as the GitHub-wide denominator, and GraphQL applying 2-axis AI judgement to a monthly random sample of N=2,000. Source data remains under its original rights; we only aggregate and visualize.
SOURCES Source list
| Source | Use | Cadence / rate | URL |
|---|---|---|---|
| GitHub GraphQL API | Repo details, owner info, signature file existence, README fetch | daily, 5,000 req/h | docs↗ |
| GitHub REST Search | Recent 24-48h new repo discovery | daily, 30 req/min | docs↗ |
| GitHub REST Code Search | AI tool config-file counts across all GitHub (reference scale) | monthly, 9 req/min | docs↗ |
| GH Archive (BigQuery) | GitHub-wide monthly event aggregates (used as denominator): create / push / fork / watch | monthly, within 1TiB/mo free tier | gharchive.org↗ |
| Anthropic API (Claude Sonnet 5.5) | Monthly report drafts only (OpenAI gpt-5-mini up to the 2026-09 report) | $3/mo budget brake | anthropic.com↗ |
| Google Trends | Search interest for related keywords (auxiliary signal) | monthly, manual CSV import | trends↗ |
PIPELINE From source to ranking
1. Discovery
Split-query GitHub REST Search (3 query groups, 3 keywords each) finds new repos matching AI-related terms. Inserted into `github_repositories`.
2. Detail fetch
GitHub GraphQL fetches repo details, owner, README, topics, and 7 AI tool signature files in a single query (batch size 20, with 5xx retry).
3. Scoring
AI relevance, solo dev, web launch scores computed monthly into `repo_metric_snapshots`. AI tool signatures into `repo_ai_signals`.
4. AI summarization
README summarisation with OpenAI gpt-5-mini stopped after 1,633 repositories and is not running for newly tracked ones. The relevance breakdown and cohort position shown on each repository page are computed from stored data and do not depend on it.
5. Monthly aggregate
GH Archive (BigQuery) → `global_monthly_aggregates` stores GitHub-wide active repo totals, used as denominator for AI involvement estimates.
5.5. 2-axis sample judgement
Deterministic random sample (FARM_FINGERPRINT, N=2,000) from GH Archive in BigQuery. GraphQL judges is_ai_built / is_ai_using; results saved to `repo_samples`. Multiplied by the denominator above to estimate GitHub-wide AI involvement. ±1pt at 95% CI.
6. Adoption count
13 category keyword dictionaries are substring-matched against each repo's corpus (description + topics + summary). Unique repo counts cached 30 min.
COST Cost & quota management
BigQuery
Month-partitioned tables + column pruning + maximum_bytes_billed=100GB hard cap keep usage within the 1TiB/mo free tier (typical: a few hundred GB).
Claude API
Monthly budget $3 USD. Before each call the month-to-date spend plus the cost of the full output ceiling is checked; a call that could breach the cap throws BudgetExceededException. Actual usage is recorded per call and per model.
GitHub API
Personal Access Token (Classic, no scopes): 5,000 core / 30 search / 9 code-search per minute. Batched + exponential-backoff retry on 5xx.
Hosting
Laravel 13 + PHP 8.5 + MySQL 5.7 on Xserver shared hosting. Monthly partitioning keeps DB size in check.