AH

Crime Data Pipeline

All-in-one: setup → source → process → results → audit

1Setup
2Sources
3Process
4Results
5Audit
Base File — Previous Month's Combined CSV

Click or drag & drop your base CSV

ori, Agency Name, Murder…, Year, Month, State, Type

Agency Sourcing

Or drag/click to upload manually

Date Range
CDE CDE / Colab Output
or upload manually

Click or drag & drop CDE CSV (Colab output or CDE Processor output)

SCRAPE Scrape CSV — Pre-Aggregated Output
or upload manually

Click or drag & drop aggregated scrape CSV (output from manual parser tool)

Agency Name, Murder, Rape, …, Year, Month

SCRAPE Auto-Scrape APIs (alternative / supplement to CSV above)
EMAIL Fetch Email Submissions

Download new RTCI submissions from Gmail to the Manual Uploads folder. Requires the email fetcher server: python3 fetch-rtci-attachments.py --serve

MANUAL Load Manual Upload Files

Select the Manual Uploads folder to auto-detect and parse files. Files are matched by name: Spokane*.xlsx, *city*.csv / *county*.csv (Ohio), Wisconsin*.csv, Miami*.xlsx, *SanJose* / *SJPD* (.html).

Or upload individual files manually…
PDF Manual Sources

Preloaded data from agency PDFs and dashboards. Upload newer PDFs to replace, or update preloaded values when new months complete.

ARTIFACT North Carolina BCP Files
or upload BCP files manually

Upload 3 BCP files from NC SBI NIBRS extract. Agency.bcp is built-in (577 agencies). Streams large files — no memory issues.

Incident.bcp

ORI+OCA → date

Offense.bcp

Property crimes

VictimOffense.bcp

Violent crimes

Pipeline
① Extend Base
② Match Sources
③ Fill / Overwrite
④ Property Metrics
⑤ Z-Scores
⑥ Flag Problems
Ready
Fill Summary
Filled Rows Preview
Still Blank (sample)
⚠ Problem Agencies

Agencies with property z-score below −2 in the most recent 12 months

Completeness Analytics
Agency Completeness
of agencies
Population Coverage
of population
Murder-Reported Months Distribution
Agency Details
Monthly Offense Totals — Complete Agencies
Year-to-Date Comparison — Complete Agencies
Potential Outliers
Upload Audited Monthly Sample

Upload the fully audited and corrected Combined CSV. All analytics below will run on the audited data.

Click or drag & drop audited CSV

Same format as the exported Combined CSV

Population Data

Pulls latest population per ORI from S3 CDE data. Requires Agency Sourcing to be loaded (for ORI lookup).