emfurst
2cd7aa1c9d
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01X9jG4CUFcH6n4vFvgtwndY
79 lines
4.5 KiB
Markdown
79 lines
4.5 KiB
Markdown
# Admin Analytics
|
|
|
|
University administrative cost benchmarking project using public data (IRS 990, IPEDS, BLS CPI-U). **First iteration is scoped to the University of Delaware only.** Peer/AAU/multi-institution comparisons are planned for a later iteration.
|
|
|
|
## Project status
|
|
|
|
Phase 1 (data acquisition) and the dashboard are built and deployed at adminalytics.org. The CLI ingests IPEDS, IRS 990, BLS CPI-U, EADA athletics, and a staff-directory scrape into DuckDB; the Dash dashboard has tabs for overview, compensation, staffing, current headcount, endowment, philanthropy, athletics, and about. Next planned iteration: peer/AAU institution comparisons.
|
|
|
|
## Architecture
|
|
|
|
- **Language:** Python
|
|
- **Database:** DuckDB (single-file, zero-config). The dashboard opens it read-only; migrating to PostgreSQL is an option if concurrent writes become necessary.
|
|
- **Package manager:** uv
|
|
- **CLI framework:** typer
|
|
- **Dashboard:** Dash + Plotly, served by gunicorn in production
|
|
- **Testing:** pytest
|
|
|
|
## Data sources
|
|
|
|
| Source | Format | What we extract |
|
|
|--------|--------|-----------------|
|
|
| IRS 990 bulk XML | XML (versioned schemas) | Filing financials, Part VII compensation, Schedule J detailed compensation |
|
|
| IPEDS | CSV bulk downloads | Institution directory (HD), finance by function (F1A/F2), staffing (S/SAL), enrollment (EF) |
|
|
| EADA (Equity in Athletics) | Excel via ope.ed.gov/athletics API | Athletics expenses by sport, coaching salaries, participants (keyed by UNITID, FY2007+) |
|
|
| BLS CPI-U | Flat file or API | Consumer Price Index for inflation-adjusted compensation analysis |
|
|
| Admin office web pages | HTML scraping | Staff directory headcounts |
|
|
|
|
## Key concepts
|
|
|
|
- **University of Delaware** is the sole target institution for the first iteration. UD's IPEDS UNITID is the anchor for all IPEDS queries.
|
|
- **UD files its own IRS 990** despite being a public university (EIN 516000297, tax years 2015+). UD's filings are the source of all executive compensation data (Schedule J), including head coaches and the athletic director. The **UD Research Foundation** (EIN 516017306) also files a 990; it contributes filing-level financials but no Schedule J rows. Philanthropy data comes from IPEDS F2, not the 990s.
|
|
- **UNITID** is the canonical institution identifier (from IPEDS). All cross-source linking flows through UNITID.
|
|
- **EIN** links to IRS 990 filings. For the first iteration, only the UD and UD Research Foundation EINs (`UD_EINS` in `config.py`) are needed. A broader UNITID-to-EIN crosswalk will be built when expanding to peer institutions.
|
|
- IRS 990 XML schemas change across tax years. Use the Master Concordance File or IRSx library to handle XPath variations.
|
|
- IPEDS variable names change across years. Always parse the accompanying data dictionary; never hardcode variable names.
|
|
|
|
## Project structure
|
|
|
|
```
|
|
src/admin_analytics/
|
|
config.py
|
|
cli.py
|
|
db/ # DuckDB schema and connection
|
|
irs990/ # 990 download, XML parsing, Schedule J extraction, university filtering
|
|
ipeds/ # IPEDS download, dictionary parsing, finance/HR/enrollment loading
|
|
bls/ # CPI-U fetcher and loader
|
|
eada/ # EADA athletics download and loader
|
|
scraper/ # Admin office staff-directory headcount scraper
|
|
dashboard/ # Dash app (app.py), queries, components, pages/, assets/, wsgi.py
|
|
validation.py # Data validation checks
|
|
data/raw/ # Downloaded files (gitignored)
|
|
tests/
|
|
fixtures/ # Sample XML/CSV files for tests
|
|
```
|
|
|
|
## Build & run
|
|
|
|
```
|
|
uv run admin-analytics ingest ipeds --year-range 2005-2024
|
|
uv run admin-analytics ingest irs990 --year-range 2019-2025
|
|
uv run admin-analytics ingest cpi
|
|
uv run admin-analytics ingest eada --year-range 2007-2025
|
|
uv run admin-analytics ingest scrape
|
|
uv run admin-analytics ingest all
|
|
uv run admin-analytics validate
|
|
uv run admin-analytics dashboard # dev server on 127.0.0.1:8050
|
|
uv run gunicorn -w 2 -b 127.0.0.1:8050 admin_analytics.dashboard.wsgi:server
|
|
uv run pytest
|
|
```
|
|
|
|
Restart the dashboard after ingesting; it holds read-only DuckDB connections.
|
|
|
|
## Conventions
|
|
|
|
- Raw data tables are prefixed with `raw_` (e.g., `raw_institution`, `raw_990_schedule_j`)
|
|
- Downloaded files go in `data/raw/` and are gitignored
|
|
- IPEDS variables are mapped to canonical column names at ingest time; raw CSVs stay on disk for reprocessing
|
|
- First iteration filters all data to UD/UD Foundation only. Design parsers to accept institution filters so they can scale to multi-institution in a later iteration
|
|
- 990 downloads are filtered by EIN from index files to avoid downloading the full archive (hundreds of GB)
|