DATA ENGINEERING
Municipal schemas become one canonical event spine.
CrimeSense models how reported crime intensity changes across space and time using a leakage-safe event spine, sampled point-process support, and spatial, temporal, environmental, infrastructural, and socioeconomic context.
CrimeSense is the public analytical system built on CrimeNet's current 15-jurisdiction data and ML pipeline: immutable lineage, canonical geography, leakage-safe features, explicit eligibility, and GPU-backed analytical delivery.
Municipal schemas become one canonical event spine.
Every cell and timestamp receives spatially aligned context.
Point-process objectives estimate event intensity, not deterministic outcomes.
Coverage is established before an estimate can be displayed.
Analytical geometry flows into WebGL instead of React DOM nodes.
The current system spans immutable S3/Parquet snapshots, a canonical event spine, exposure-weighted integration sampling, point-in-time feature enrichment, XGBoost training, a FastAPI serving boundary, and the GPU Explorer.
Municipal + county records · OSM · weather · ACS · solar
S3 · Delta / Parquet · immutable snapshot lineage
Bronze → Silver → leakage-safe Gold event spine
H3 support sampling · exposure weights · temporal support
Point-in-time H3 · environment · causal event history
XGBoost point process · geographic CV · GPU acceleration
FastAPI · Next.js · MapLibre · deck.gl GPU surface
CrimeSense uses a Bronze–Silver–Gold data contract, but the current pipeline is no longer coupled to one managed compute platform. Dagster, Polars, S3, Delta/Parquet, DuckDB, Python, and SQL carry the active snapshot and feature workflow.
More than 17 million municipal and county crime records have been ingested across the expanded source footprint, with source identity, lineage, and time semantics preserved before canonicalization.
_ingestion_run_id · _ingested_at_utcOffense mappings, units, timestamps, H3 identifiers, OSM values, and socioeconomic periods are validated and projected into stable schemas.
source schema → canonical event contractThe current point-process dataset expands observed events with exposure-weighted integration support to more than 180 million model examples, enriched with temporal, weather, lighting, OSM, ACS, and causal history context.
event rows · integration rows · exposure weightsOffline scores are meaningless when future state leaks into historical examples. CrimeSense constructs causal history strictly before each model timestamp, freezes support by split, and keeps the 2025+ test partition sealed during training and validation.
FEATURE TIME ≤ PREDICTION TIMEH3 is an architectural primitive across spatial joins, aggregation, neighborhoods, model observations, inference, and rendering. The Explorer renders backend-selected H3-r4 through H3-r9 LOD cells derived from one canonical r9 inference surface.
Canonical temporal-history columns in the event-spine contract, built strictly from prior events
Local hour and day-of-week with cyclical encodings
Temperature, humidity, availability, solar geometry, daylight, and lighting state
POI, road, intersection, building, land-use, and urban-mix densities
ACS population, income, age, poverty, employment, housing, tenure, and mobility context
The current baseline uses XGBoost with a Poisson point-process objective over exposure-weighted integration samples. The active geographic-CV baseline uses 37 numeric features plus one categorical lighting feature; richer causal-history columns remain available in the final model contract.
TEST ACCESS: FALSECurrent geographic-CV runs use deterministic training and validation samples across five held-out geographic folds. These bars show the global split policy; each source is further clipped to its documented temporal support. The test split remains untouched during model selection and is reserved for final evaluation.
A geographic coordinate is not automatically a valid model input. CrimeSense establishes required feature state before displaying intensity; the current event-spine audit reports 99.969% modeled coverage while preserving missingness explicitly.
lat · lon · timeMapLibre owns the geographic camera and label stack. deck.gl interleaves H3 geometry beneath those labels, keeping thousands of cells in GPU-backed layers instead of React DOM.
AUG 21, 2024 · 17:00 CDTCrimeSense normalizes heterogeneous municipal and county records into one event contract while preserving source identity and local time. The current event spine spans 15 jurisdictions and 74,689 unique H3 cells; support still does not imply complete covariate coverage for every cell-time.
The inventory below reflects imported dependencies and implemented code paths—not an aspirational logo cloud.
CrimeNet Ω remains a research track rather than the selected serving baseline. It explores neural marked point-process formulations over the same leakage-safe spatial, temporal, and contextual contracts.
Neural history, graph structure, Hawkes-style excitation, multiscale state, and reporting-process components remain research directions. They are not presented as properties of the current XGBoost serving baseline.
CrimeSense's strongest guarantees are the ones that prevent an attractive interface from overstating what the data and model can support.
Historical predictions can only use information established before their model timestamp.
Heterogeneous locations are normalized into stable H3 cells and authoritative jurisdiction masks.
Versioned configurations, deterministic samples, run IDs, and artifacts preserve experiment identity.
Missing feature coverage can never masquerade as zero intensity or a safe region.
Lazy columnar scans, partitioned Delta tables, and GPU rendering keep large work off the DOM.
Runtime schemas protect the interface from malformed or semantically invalid inference responses.
Inspect the model surface, temporal state, and coverage contract directly.