IncidentLens

IncidentLens detects city-scale incidents from the enriched observation stream produced by Urban Observation Processing. It maintains competing hypotheses over space and time, then writes incident predictions and optionally updates a live map.

The implementation paper is in docs/incidentlens.pdf. The static project site is in docs/ and can be published with GitHub Pages.

How the pieces fit together

Component	Purpose	Required?
Urban Observation Processing	Receives observations, applies shared anomaly gating and enrichment, and writes the durable common stream	Yes
IncidentLens pipeline	Adapts the common stream and performs detection	Yes
IncidentLens visualization	Displays observations and predictions at http://127.0.0.1:8000	No
IncidentLens Benchmarks	Owns private labels and evaluates detector results	No

Observation replay sender
          |
          v
receiver + shared enrichment ---> observations.jsonl
                                         |
                                         v
                              IncidentLens detection ---> results
                                         |
                                         +---> optional map

IncidentLens does not run a second receiver and does not read raw archives. It follows the enriched JSONL stream using its own restart-safe byte offset. Real and synthetic observations use the same schema and are recognized automatically.

Shared annotations are authoritative. IncidentLens does not repeat routine LLM, VLM, geocoding, or anomaly work. It may request forced enrichment through the same shared service when an observation is inside an active hypothesis.

Quick start with Docker

Requirements:

• Docker with Compose
• urban-observation-processing and incidentlens cloned as sibling directories
• an OpenAI API key

Create the private configuration:

cd incidentlens
cp config.example.json config.json
chmod 600 config.json

Edit config.json, especially openai.api and the writable paths. The file is ignored by Git.

Start the shared processing services:

cd ../urban-observation-processing
./docker-start up

Then start IncidentLens:

cd ../incidentlens
./docker-start up

To include the optional visualization server, use:

./docker-start visual

Open http://127.0.0.1:8000/. The wrappers connect both repositories through the urban-observations-processing Docker network and derive mounts from their configuration files.

IncidentLens needs one routine command: ./docker-start up for detection, or ./docker-start visual to start detection plus the optional map. These are alternatives, not two required commands. A new checkout uses three commands for configuration and startup (cp, chmod, and one wrapper command). Shared processing is a separate deployment whose command count is documented in its own repository.

Configuration

The commands read ./config.json by default. Start from config.example.json.

Setting	Meaning
openai.api	API key used by the full model
processing.enrichment_url	Shared enrichment endpoint
processing.enriched_stream_path	Durable enriched JSONL stream
detection.incident_types	Text file containing one permitted incident type per line
detection.baseline	Detector implementation; normally incidentlens
detection.follow_input	Wait for new complete records when the stream reaches EOF
visualization.enabled	Send live layers to the optional map server
paths.*	Writable runtime, result, and cache directories

The cursor is managed automatically under paths.runtime_root/state. It advances only after successful processing and resets safely if the stream is replaced. Command-line flags are available as temporary overrides; normal use requires no pipeline arguments.

Local Python alternative

Python 3.10 or newer is required. Keep IncidentLens in its own virtual environment:

python3.10 -m venv .venv
. .venv/bin/activate
python -m pip install --upgrade pip setuptools wheel
python -m pip install -e ../urban-observation-processing
python -m pip install -r requirements.txt
python -m pip install --no-deps -e .
python -m detection.full_pipeline

The editable processing dependency provides the shared data model and enrichment client; it does not start collectors. To check an installation without running models, use python -m detection.full_pipeline --help.

Synthetic replay

urban-observation-simulator generates completed datasets, urban-observation-processing replays and enriches them, and incidentlens-benchmarks keeps private labels and evaluates predictions. The opaque synthetic run ID separates results without exposing ground truth to the detector. IncidentLens requires no synthetic-specific mode or configuration.

Outputs

The main outputs are:

• low_level_results.json for accepted low-level predictions
• high_level_results.json for incident compositions, when enabled
• timing JSON/JSONL files
• hypotheses and candidate incidents under paths.runtime_root

Results are stored below paths.results_root; real and synthetic runs are kept separate. Evaluation and authoritative labels remain in incidentlens-benchmarks.

Troubleshooting

• No module named processing: install the sibling Urban Observation Processing package as shown above.
• Configuration error: copy config.example.json to config.json and fill in the API key and writable paths.
• No map updates: use ./docker-start visual and confirm visualization.enabled is true.