diff --git a/CLAUDE.md b/CLAUDE.md index 7fdba2e..07a1b8d 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -4,14 +4,15 @@ University administrative cost benchmarking project using public data (IRS 990, ## Project status -Currently in planning. Phase 1 (Data Acquisition) is planned but not yet built. See `phase1_plan.md` for the full implementation plan and `administrative_analytics_scope_v0.1.md` for project scope. +Phase 1 (data acquisition) and the dashboard are built and deployed at adminalytics.org. The CLI ingests IPEDS, IRS 990, BLS CPI-U, EADA athletics, and a staff-directory scrape into DuckDB; the Dash dashboard has tabs for overview, compensation, staffing, current headcount, endowment, philanthropy, athletics, and about. Next planned iteration: peer/AAU institution comparisons. ## Architecture - **Language:** Python -- **Database:** DuckDB for Phase 1 (single-file, zero-config). Migrate to PostgreSQL in Phase 3 when the dashboard needs concurrent access. +- **Database:** DuckDB (single-file, zero-config). The dashboard opens it read-only; migrating to PostgreSQL is an option if concurrent writes become necessary. - **Package manager:** uv -- **CLI framework:** typer or click (TBD) +- **CLI framework:** typer +- **Dashboard:** Dash + Plotly, served by gunicorn in production - **Testing:** pytest ## Data sources @@ -22,7 +23,7 @@ Currently in planning. Phase 1 (Data Acquisition) is planned but not yet built. | IPEDS | CSV bulk downloads | Institution directory (HD), finance by function (F1A/F2), staffing (S/SAL), enrollment (EF) | | EADA (Equity in Athletics) | Excel via ope.ed.gov/athletics API | Athletics expenses by sport, coaching salaries, participants (keyed by UNITID, FY2007+) | | BLS CPI-U | Flat file or API | Consumer Price Index for inflation-adjusted compensation analysis | -| Admin office web pages (stretch) | HTML scraping | Staff directory headcounts | +| Admin office web pages | HTML scraping | Staff directory headcounts | ## Key concepts @@ -33,7 +34,7 @@ Currently in planning. Phase 1 (Data Acquisition) is planned but not yet built. - IRS 990 XML schemas change across tax years. Use the Master Concordance File or IRSx library to handle XPath variations. - IPEDS variable names change across years. Always parse the accompanying data dictionary; never hardcode variable names. -## Planned project structure +## Project structure ``` src/admin_analytics/ @@ -43,7 +44,10 @@ src/admin_analytics/ irs990/ # 990 download, XML parsing, Schedule J extraction, university filtering ipeds/ # IPEDS download, dictionary parsing, finance/HR/enrollment loading bls/ # CPI-U fetcher and loader - scraper/ # Stretch: admin office headcount scraper + eada/ # EADA athletics download and loader + scraper/ # Admin office staff-directory headcount scraper + dashboard/ # Dash app (app.py), queries, components, pages/, assets/, wsgi.py + validation.py # Data validation checks data/raw/ # Downloaded files (gitignored) tests/ fixtures/ # Sample XML/CSV files for tests @@ -51,15 +55,21 @@ tests/ ## Build & run -Not yet implemented. When built, the CLI will support: ``` -admin-analytics ingest ipeds --year-range 2005-2024 -admin-analytics ingest irs990 --year-range 2005-2024 -admin-analytics ingest cpi -admin-analytics ingest eada --year-range 2007-2025 -admin-analytics ingest all +uv run admin-analytics ingest ipeds --year-range 2005-2024 +uv run admin-analytics ingest irs990 --year-range 2019-2025 +uv run admin-analytics ingest cpi +uv run admin-analytics ingest eada --year-range 2007-2025 +uv run admin-analytics ingest scrape +uv run admin-analytics ingest all +uv run admin-analytics validate +uv run admin-analytics dashboard # dev server on 127.0.0.1:8050 +uv run gunicorn -w 2 -b 127.0.0.1:8050 admin_analytics.dashboard.wsgi:server +uv run pytest ``` +Restart the dashboard after ingesting; it holds read-only DuckDB connections. + ## Conventions - Raw data tables are prefixed with `raw_` (e.g., `raw_institution`, `raw_990_schedule_j`)