Skip to content

Commit 527f165

Browse files
committed
refactor: unify attribution into manifest dataSources and remove per-item attribution objects
Replace inline attribution objects on DataSourceDetail, SharedMobilityStation, SharedMobilityVehicle, and weather responses with a sourceId-based lookup system that resolves attribution from integration manifest dataSources at render time. This eliminates hardcoded attribution objects scattered across providers and consolidates all licensing/attribution metadata into the single source of truth established by the dataSources manifest schema. Core type changes: - DataSourceDetail: replace `source: string` with `sources: string[]`, remove `attribution` field - DataSourceMeta: remove `id`, `name`, `attribution`, `categoryChipLabel` (now sourced from manifest) - DataSourceProvider.getDetail() now returns `DataSourceDetail | null` instead of synthetic fallback objects - DataSourceFilterDef: add `clientSide?: boolean` flag for filters applied locally instead of sent to API - DataSourceDetailSection: add `image` and `embed` section types with imageUrl/embedUrl/embedType fields, add `videocam` and `warning` section icons - SharedMobilityStation/Vehicle: replace `source: string` + `attribution` with `sources: string[]` - IntegrationDataSource: add required `sourceId` field to all ~90 manifest dataSources entries - WeatherResponse: remove inline `attribution` object, use `source` field for registry lookup Attribution rendering: - Add `buildSourceAttribution()` and `extractSourcePrefix()` utilities to packages/core - DataSourceSections: new AttributionFooter component resolves attribution from integration registry by matching source prefixes against manifest dataSources - DataSourceLayer: build map attribution from visible result sources via registry lookup, recreate MapLibre source when attribution HTML changes - PlaceWeather/WeatherWidget: resolve weather attribution from integration registry instead of inline object - PhotoAttribution/PlacePhotoGallery: resolve photo source names and URLs from registry instead of hardcoded PHOTO_SOURCES map - DataSourceFilterContent/DataSourceLayer: replace duplicated CLIENT_SIDE_FILTER_IDS set with shared `splitFilters()` and `applyClientSideFilters()` utilities using the new `clientSide` filter flag Data source orchestrator: - listWithFilters() now derives id, name, categoryChipLabel from integration manifest instead of provider meta - data-source detail endpoint returns 404 when provider returns null instead of serving empty fallback Provider cleanup (bike-sharing, car-sharing, ev-charging, fuel, scooter-sharing, parking): - Remove all inline attribution objects from station/vehicle/result construction - Remove fallback detail objects on cache miss (return null) - Remove server-side speed filter from ev-charging (now client-side only) - merge-stations: merge source arrays instead of attribution objects, remove mergeAttributions import SSRF protection: - Consolidate three duplicate URL validation implementations (gtfs/importer.ts, validate-url.ts, store.ts) into shared `validatePublicUrl()` from @openmapx/core - Delete apps/api/src/utils/validate-url.ts - Add validatePublicUrl() call to catalog source fetching Parking integration expansion: - Add 12 new parking data sources to manifest: RDW Netherlands, BNLS France, Ghent, Brussels, Basel, Florence, Barcelona, Vienna, Copenhagen, Singapore, Madrid, UTMC Newcastle, NSW Australia - Add corresponding health checks, env vars (UTMC_USERNAME/PASSWORD, NSW_TRANSPORT_API_KEY, WINDY_WEBCAMS_API_KEY), and i18n strings - Add new source headers for webcam sources (windy, osm-webcam, caltrans, tfl) Opening hours: - Handle unknown/unparseable opening hours gracefully (isUnknown flag) in PlaceOverviewTab and CategoryResultsContent instead of showing misleading open/closed status Other fixes: - PlaceDetailContent: use `bgcolor: "background.paper"` for sticky tabs instead of `inherit` - CyclingBaseLayer: fix OSM attribution link text - Overlay attribution text corrections (hiking, winter-sports)
1 parent c391f65 commit 527f165

171 files changed

Lines changed: 5420 additions & 1320 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

apps/api/.env.example

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -86,6 +86,10 @@ FIRMS_MAP_KEY=
8686
# Register at https://openchargemap.org/ and get an API key
8787
OPENCHARGEMAP_API_KEY=
8888

89+
# Webcams: Windy Webcams API
90+
# Get a free API key at https://api.windy.com/keys
91+
WINDY_WEBCAMS_API_KEY=
92+
8993
# Fuel prices: Tankerkoenig (Germany only)
9094
# Get a free API key at https://creativecommons.tankerkoenig.de/
9195
TANKERKOENIG_API_KEY=
@@ -101,6 +105,15 @@ DB_RIS_API_KEY=
101105
DB_PARKING_CLIENT_ID=
102106
DB_PARKING_API_KEY=
103107

108+
# Parking: UTMC Tyne & Wear (Newcastle, UK)
109+
# Request credentials by emailing data@glasgow.gov.uk / NE Travel Data portal
110+
UTMC_USERNAME=
111+
UTMC_PASSWORD=
112+
113+
# Parking: Transport for NSW (Sydney, Australia)
114+
# Register at https://opendata.transport.nsw.gov.au/ for a free API key
115+
NSW_TRANSPORT_API_KEY=
116+
104117
# Deutsche Bahn: Shared Mobility GBFS (Call-a-Bike / StadtRad)
105118
# Register at https://developers.deutschebahn.com/ and subscribe to the Shared Mobility GBFS API
106119
DB_GBFS_CLIENT_ID=

apps/api/src/services/gtfs/importer.ts

Lines changed: 2 additions & 36 deletions
Original file line numberDiff line numberDiff line change
@@ -3,46 +3,12 @@ import { createHash } from "node:crypto";
33
import { existsSync, mkdirSync, readdirSync, readFileSync, rmSync, writeFileSync } from "node:fs";
44
import { tmpdir } from "node:os";
55
import { join } from "node:path";
6+
import { validatePublicUrl } from "@openmapx/core";
67
import { gtfsDate, parseCsv, streamCsvBatches } from "./csv";
78
import { sql } from "./db";
89

910
const BATCH_SIZE = 5_000;
1011

11-
// URL Validation
12-
13-
const PRIVATE_IP_RANGES = [
14-
/^127\./, // loopback
15-
/^10\./, // 10.0.0.0/8
16-
/^172\.(1[6-9]|2\d|3[01])\./, // 172.16.0.0/12
17-
/^192\.168\./, // 192.168.0.0/16
18-
/^169\.254\./, // link-local (AWS metadata)
19-
/^0\./, // 0.0.0.0/8
20-
/^100\.(6[4-9]|[7-9]\d|1[01]\d|12[0-7])\./, // 100.64.0.0/10 (CGNAT)
21-
/^::1$/, // IPv6 loopback
22-
/^f[cd]/, // IPv6 private
23-
/^fe80:/, // IPv6 link-local
24-
];
25-
26-
function validateDownloadUrl(url: string): void {
27-
let parsed: URL;
28-
try {
29-
parsed = new URL(url);
30-
} catch {
31-
throw new Error("Invalid URL");
32-
}
33-
if (parsed.protocol !== "https:" && parsed.protocol !== "http:") {
34-
throw new Error("Only HTTP(S) URLs are allowed");
35-
}
36-
const hostname = parsed.hostname;
37-
if (
38-
hostname === "localhost" ||
39-
hostname === "" ||
40-
PRIVATE_IP_RANGES.some((re) => re.test(hostname))
41-
) {
42-
throw new Error("URLs targeting internal/private addresses are not allowed");
43-
}
44-
}
45-
4612
// Schema DDL
4713

4814
function createSchemaDDL(schema: string): string {
@@ -191,7 +157,7 @@ function createServiceDaysDDL(schema: string): string {
191157
// Download & Extract
192158

193159
function downloadAndExtract(url: string, tempDir: string): string {
194-
validateDownloadUrl(url);
160+
validatePublicUrl(url);
195161
mkdirSync(tempDir, { recursive: true });
196162
const zipPath = join(tempDir, "feed.zip");
197163

apps/api/src/services/gtfs/index.ts

Lines changed: 1 addition & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,4 @@
1-
import type { BBox } from "@openmapx/core";
2-
import { validatePublicUrl } from "../../utils/validate-url";
1+
import { type BBox, validatePublicUrl } from "@openmapx/core";
32
import { sql } from "./db";
43
import { dropGtfsSchema, importGtfsFeed } from "./importer";
54
import type { CatalogFeed, FeedStatus, ImportedFeed } from "./types";

apps/api/src/services/motis/manager.ts

Lines changed: 1 addition & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -5,9 +5,8 @@ import { promisify } from "node:util";
55
import { motisLocalInstance } from "@integrations/transit-motis/instances.js";
66
import type { MotisFeed, MotisStatus } from "@integrations/transit-motis/types";
77
import { stops } from "@motis-project/motis-client";
8-
import type { BBox } from "@openmapx/core";
8+
import { type BBox, validatePublicUrl } from "@openmapx/core";
99
import { cacheGet, cacheSet } from "../../utils/cache.js";
10-
import { validatePublicUrl } from "../../utils/validate-url";
1110

1211
const execFileAsync = promisify(execFile);
1312

apps/api/src/services/store.ts

Lines changed: 8 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,12 @@ import { spawn } from "node:child_process";
22
import { existsSync, readFileSync } from "node:fs";
33
import { dirname, join } from "node:path";
44
import { fileURLToPath } from "node:url";
5-
import { PLATFORM_VERSION, satisfiesPlatformVersion, USER_AGENT_ADMIN } from "@openmapx/core";
5+
import {
6+
PLATFORM_VERSION,
7+
satisfiesPlatformVersion,
8+
USER_AGENT_ADMIN,
9+
validatePublicUrl,
10+
} from "@openmapx/core";
611
import { eq } from "drizzle-orm";
712
import { db } from "../db";
813
import { installedIntegration } from "../db/schema";
@@ -66,23 +71,14 @@ async function getExtraSources(): Promise<CatalogSource[]> {
6671
export async function addCatalogSource(url: string, label: string): Promise<void> {
6772
if (!redis) return;
6873

74+
validatePublicUrl(url);
6975
const parsed = new URL(url);
7076
if (parsed.protocol !== "https:") {
7177
throw new Error("Catalog source must use HTTPS");
7278
}
7379
if (parsed.username || parsed.password) {
7480
throw new Error("Catalog source URL must not contain credentials");
7581
}
76-
const hostname = parsed.hostname.toLowerCase();
77-
if (
78-
hostname === "localhost" ||
79-
hostname.endsWith(".local") ||
80-
hostname.endsWith(".internal") ||
81-
/^(127\.|10\.|172\.(1[6-9]|2\d|3[01])\.|192\.168\.|169\.254\.|0\.)/.test(hostname) ||
82-
hostname === "[::1]"
83-
) {
84-
throw new Error("Catalog source must not point to internal addresses");
85-
}
8682

8783
const existing = await getExtraSources();
8884
if (existing.some((s) => s.url === url)) return;
@@ -106,6 +102,7 @@ export async function listCatalogSources(): Promise<CatalogSource[]> {
106102
}
107103

108104
async function fetchCatalogFromUrl(url: string): Promise<CatalogEntry[]> {
105+
validatePublicUrl(url);
109106
const res = await fetch(url, {
110107
headers: { "User-Agent": USER_AGENT_ADMIN },
111108
signal: AbortSignal.timeout(15_000),

apps/web/src/components/map/layers/CyclingBaseLayer.tsx

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -11,7 +11,7 @@ const OSM_ATTRIBUTION = buildAttributionHtml({
1111
license: "CC-BY-SA",
1212
licenseUrl: "https://creativecommons.org/licenses/by-sa/2.0/",
1313
attribution:
14-
'© <a href="https://www.openstreetmap.org/copyright" target="_blank" rel="noopener noreferrer">OpenStreetMap</a> contributors (<a href="https://creativecommons.org/licenses/by-sa/2.0/" target="_blank" rel="noopener noreferrer">CC-BY-SA</a>)',
14+
'© <a href="https://www.openstreetmap.org/copyright" target="_blank" rel="noopener noreferrer">OpenStreetMap contributors</a> (<a href="https://creativecommons.org/licenses/by-sa/2.0/" target="_blank" rel="noopener noreferrer">CC-BY-SA</a>)',
1515
});
1616

1717
const CYCLOSM_ATTRIBUTION = [

apps/web/src/components/map/layers/DataSourceLayer.tsx

Lines changed: 51 additions & 103 deletions
Original file line numberDiff line numberDiff line change
@@ -1,33 +1,29 @@
11
"use client";
22

3-
import type {
4-
DataSourceAttribution,
5-
DataSourceMeta,
6-
DataSourceResult,
7-
LngLat,
8-
} from "@openmapx/core";
3+
import type { DataSourceMeta, DataSourceResult, LngLat } from "@openmapx/core";
94
import {
5+
applyClientSideFilters,
6+
buildSourceAttribution,
107
PANEL,
8+
splitFilters,
119
useDataSourceSearch,
1210
useDataSourceStore,
1311
useDataSources,
12+
useIntegrationRegistry,
1413
useOpeningHoursStore,
1514
usePlaceStore,
1615
useSidebarStore,
1716
} from "@openmapx/core";
1817
import type maplibregl from "maplibre-gl";
1918
import type { GeoJSONSource, Map as MaplibreMap, MapMouseEvent } from "maplibre-gl";
20-
import { useCallback, useEffect, useMemo, useRef, useState } from "react";
19+
import { useCallback, useEffect, useMemo, useRef } from "react";
2120
import { usePinMarker } from "@/hooks/usePinMarker";
2221
import { INTERACTIVE_LAYER_IDS } from "@/lib/interactiveLayers";
2322
import { useMap } from "@/lib/MapContext";
2423
import { createMarkerSvg } from "@/lib/markerSvg";
2524
import { getFirstSymbolLayerId } from "./layerStyleUtils";
2625
import { useLayerReanchor } from "./useLayerReanchor";
2726

28-
/** Filter IDs that are applied client-side instead of being sent to the API. */
29-
const CLIENT_SIDE_FILTER_IDS = new Set(["operator", "speed"]);
30-
3127
function sourceId(dsId: string) {
3228
return `ds-${dsId}`;
3329
}
@@ -40,21 +36,7 @@ function labelsLayerId(dsId: string) {
4036
return `ds-${dsId}-labels`;
4137
}
4238

43-
function buildGeoJson(
44-
results: DataSourceResult[],
45-
attribution: DataSourceAttribution | DataSourceAttribution[],
46-
imageId?: string,
47-
) {
48-
const attrs = Array.isArray(attribution) ? attribution : [attribution];
49-
const attrHtml = attrs
50-
.map((a) => {
51-
const name = `<a href="${a.url}">${a.text}</a>`;
52-
if (!a.license) return name;
53-
const license = a.licenseUrl ? `<a href="${a.licenseUrl}">${a.license}</a>` : a.license;
54-
return `${name} (${license})`;
55-
})
56-
.join(", ");
57-
39+
function buildGeoJson(results: DataSourceResult[], attributionHtml: string, imageId?: string) {
5840
return {
5941
type: "FeatureCollection" as const,
6042
features: results.map((r) => ({
@@ -74,7 +56,7 @@ function buildGeoJson(
7456
...(imageId ? { imageId } : {}),
7557
},
7658
})),
77-
attribution: attrHtml,
59+
attribution: attributionHtml,
7860
};
7961
}
8062

@@ -157,23 +139,23 @@ export function DataSourceLayer() {
157139

158140
const { data: sourcesData } = useDataSources();
159141
const prevSourceRef = useRef<string | null>(null);
142+
const prevAttrRef = useRef<string>("");
160143

161144
// Find meta for the active source
162145
const activeMeta = useMemo(() => {
163146
if (!activeSource || !sourcesData?.sources) return null;
164147
return sourcesData.sources.find((s) => s.id === activeSource) ?? null;
165148
}, [activeSource, sourcesData]);
166149

167-
// Separate client-side vs server-side filters
168-
const serverFilters = useMemo(() => {
169-
const result: Record<string, unknown> = {};
170-
for (const [key, value] of Object.entries(filters)) {
171-
if (!CLIENT_SIDE_FILTER_IDS.has(key)) {
172-
result[key] = value;
173-
}
174-
}
175-
return result;
176-
}, [filters]);
150+
const registry = useIntegrationRegistry();
151+
152+
// Separate server-side filters (sent to the API) from client-side filters
153+
// (applied locally on the result set). Uses the `clientSide` flag from
154+
// provider filter definitions instead of a hardcoded list.
155+
const serverFilters = useMemo(
156+
() => splitFilters(filters, activeMeta?.filters ?? []).serverFilters,
157+
[filters, activeMeta?.filters],
158+
);
177159

178160
// Only fetch if zoom >= minZoom, using searchBbox (not viewportBbox)
179161
const shouldFetch =
@@ -187,71 +169,22 @@ export function DataSourceLayer() {
187169
serverFilters,
188170
);
189171

190-
// Accumulate results across searches (useState so map re-renders on new data)
191-
const [accumulatedMap, setAccumulatedMap] = useState(() => new Map<string, DataSourceResult>());
192-
const prevActiveRef = useRef(activeSource);
193-
const prevFiltersRef = useRef(serverFilters);
194-
const prevSearchBboxRef = useRef(searchBbox);
195-
196-
useEffect(() => {
197-
if (
198-
prevActiveRef.current !== activeSource ||
199-
prevFiltersRef.current !== serverFilters ||
200-
prevSearchBboxRef.current !== searchBbox
201-
) {
202-
setAccumulatedMap(new Map());
203-
prevActiveRef.current = activeSource;
204-
prevFiltersRef.current = serverFilters;
205-
prevSearchBboxRef.current = searchBbox;
206-
return;
207-
}
208-
209-
if (searchResults) {
210-
setAccumulatedMap((prev) => {
211-
const next = new Map(prev);
212-
for (const r of searchResults) next.set(r.id, r);
213-
return next;
214-
});
215-
}
216-
}, [activeSource, serverFilters, searchBbox, searchResults]);
217-
218-
const allResults = useMemo(() => Array.from(accumulatedMap.values()), [accumulatedMap]);
219-
220-
// Apply client-side operator/speed filters
221-
const filteredResults = useMemo(() => {
222-
if (allResults.length === 0) return [];
223-
224-
let results = allResults;
225-
226-
const speedFilter = filters.speed;
227-
if (speedFilter) {
228-
const speedValues = Array.isArray(speedFilter)
229-
? (speedFilter as string[])
230-
: [String(speedFilter)];
231-
if (speedValues.length > 0) {
232-
const speedSet = new Set(speedValues);
233-
results = results.filter((r) => speedSet.has(r.variant));
234-
}
235-
}
236-
237-
const operatorFilter = filters.operator;
238-
if (operatorFilter) {
239-
const operatorValues = Array.isArray(operatorFilter)
240-
? (operatorFilter as string[])
241-
: [String(operatorFilter)];
242-
if (operatorValues.length > 0) {
243-
const operatorSet = new Set(operatorValues);
244-
results = results.filter((r) => r.operator && operatorSet.has(r.operator));
245-
}
246-
}
247-
248-
// "Open now" filter from the opening hours chip
249-
if (openingHoursFilter === "open_now") {
250-
results = results.filter((r) => r.variant === "open");
251-
}
172+
// Apply client-side filters (speed, operator, opening hours) to the search
173+
// results from React Query. No manual accumulation — React Query is the
174+
// single source of truth and handles caching/staleness.
175+
const filteredResults = useMemo(
176+
() => applyClientSideFilters(searchResults ?? [], filters, openingHoursFilter),
177+
[searchResults, filters, openingHoursFilter],
178+
);
252179

253-
return results;
254-
}, [allResults, filters.speed, filters.operator, openingHoursFilter]);
180+
// Build map attribution from only the sources present in visible results
181+
const mapAttribution = useMemo(() => {
182+
if (!activeSource || filteredResults.length === 0) return "";
183+
const meta = registry.get(activeSource);
184+
if (!meta?.dataSources) return "";
185+
const visibleSources = [...new Set(filteredResults.map((r) => r.source))];
186+
return buildSourceAttribution(meta.dataSources, visibleSources);
187+
}, [activeSource, registry, filteredResults]);
255188

256189
// Show pin marker for hovered item
257190
const hoveredResult = filteredResults.find((r) => r.id === hoveredItemId) ?? null;
@@ -339,9 +272,15 @@ export function DataSourceLayer() {
339272

340273
const useIconMarkers = activeMeta.markerStyle.type === "icon";
341274
const imageId = useIconMarkers ? `ds-marker-${activeSource}` : undefined;
342-
const geojson = buildGeoJson(filteredResults, activeMeta.attribution, imageId);
275+
const geojson = buildGeoJson(filteredResults, mapAttribution, imageId);
276+
277+
// MapLibre doesn't support updating source attribution after creation,
278+
// so recreate the source when attribution changes.
279+
if (map.getSource(sid) && prevAttrRef.current !== geojson.attribution) {
280+
removeLayers(map, activeSource);
281+
}
282+
prevAttrRef.current = geojson.attribution;
343283

344-
// Update or create GeoJSON source
345284
if (map.getSource(sid)) {
346285
(map.getSource(sid) as GeoJSONSource).setData(geojson);
347286
} else {
@@ -452,7 +391,16 @@ export function DataSourceLayer() {
452391
return () => {
453392
map.off("styledata", syncLayer);
454393
};
455-
}, [activeSource, activeMeta, filteredResults, viewportZoom, mapReady, styleVersion, mapRef]);
394+
}, [
395+
activeSource,
396+
activeMeta,
397+
filteredResults,
398+
viewportZoom,
399+
mapReady,
400+
styleVersion,
401+
mapRef,
402+
mapAttribution,
403+
]);
456404

457405
const { setSelectedPlace } = usePlaceStore();
458406

0 commit comments

Comments
 (0)