wesamsabry/Airflow-ETL-Retail-Project
End-to-end ETL pipeline using Apache Airflow that extracts data from CSV and REST API, transforms and validates it, and loads it into PostgreSQL with incremental loading and monitoring.
SUMMARY AI summary by gpt-5-mini
Retail Data Pipeline — Astro Airflow Pipelines is a production-ready ETL framework that uses Apache Airflow (Astro CLI) to orchestrate end-to-end ingestion, transformation, validation and incremental loading of retail data into a PostgreSQL data warehouse. Intended for data engineers and platform teams, it demonstrates a modular, testable architecture combining CSV batch and REST API sources. Key features include idempotent upserts with a high‑watermark incremental engine, fault‑tolerant API handling (retries/backoff), schema and row‑level validation, dimensional modeling (fact/dim), structured logging and observability, Dockerized local runtime, and Airflow DAG-based orchestration.
DETECTED Detected AI stack
AI-related keywords found in this repo's description, topics, or README summary — grouped by category. Each badge links to the corresponding ranking detail page.
Owner
Dates
| Created on GitHub | 2026-05-12 |
| Last push | 2026-05-12 |
| First seen here | 2026-05-12 |
| Last fetched | 2026-05-12 16:10 |