Skip to content

Commit cff4fe0

Browse files
committed
feat(integrations): artifact-based community installs with import-map runtime
Make community integrations installable in production without rebuilding or touching the app-api image. Installs land in a bind-mounted `custom_integrations/` so they survive container replacement; backend code loads from `dist/backend/index.mjs`, frontend from `dist/frontend/index.js`. The api never builds, never `pnpm install`s, never imports plugin tooling. Admin install = artifact only Two payload shapes on POST /admin/store/install: { repository } catalog entry (artifact.url required) { artifactUrl, sha256?, version? } manual install Source installs from git remain a CLI-only developer workflow. The Install-from-URL dialog drops the Source/Artifact toggle in favour of a single artifact URL + optional sha256 form. Updates require the integration to be catalog-managed (sourceType=registry + repository match) so we always have a fresh artifact URL to install from; `canUpdateFromCatalog` is the single helper used by the listing badge, the install endpoint, and the store service. Drops `OPENMAPX_INTEGRATION_BUILDER_IMAGE` entirely (admin install never needs Docker-in-Docker to build) and drops the `git` value from the `source_type` enum — only `registry | artifact` now. CLI build path `pnpm openmapx integrations install ./src` and `... package <src> --out file.tar.gz` both bundle with esbuild's JS API (`import("esbuild")`), so the build only ever runs in the process that has esbuild installed. The installer surfaces a clear error if a non-CLI consumer asks for a build. Drops the `findEsbuildBinary` lookup-on-disk path, the NODE_PATH reconstruction, the `--builder-image` flag on every command, and the `validateCommunitySourceBoundary` regex check — esbuild's externals/aliases are the contract; disallowed imports fail at bundle time. Artifact contract Tarball ships: manifest.json dist/frontend/index.js when frontend components are declared dist/backend/index.mjs when backend code is present openmapx-artifact.json platform version + per-bundle sha256 strings/<locale>.json optional Explicitly forbidden: node_modules/. Bundles must be self-contained. Manifest drops `npmDependencies`, `frontendDependencies`, and `runtimeDependencies`. Tar extraction keeps the existing zip-slip, symlink/hardlink, and PAX-escape guards. Frontend runtime singletons via import maps Replaces the regex-scraped `window.__openmapx_runtime` shim system with a native `<script type="importmap">` in apps/web/src/app/layout.tsx that points `react`, `react/jsx-runtime`, `react/jsx-dev-runtime`, and `@openmapx/core` at prebuilt ESM files under apps/web/public/runtime/. The new apps/web/scripts/build-runtime-modules.mjs runs in `predev` and `prebuild` to emit those files via esbuild. Integration bundles keep bare `import React from "react"` as externals; the browser resolves them at runtime, so React + @openmapx/core stay strict singletons across every loaded community bundle and adding a new core export "just works" without re-packaging artifacts. Drops `coreRuntimeExportNames`, `REACT_RUNTIME_EXPORTS`, `CORE_RUNTIME_EXPORT_FALLBACKS`, `writeRuntimeShims`, and the `window.__openmapx_runtime` wiring in IntegrationProvider. Backend reload semantics In dev (`NODE_ENV !== "production"`), backend module URLs are still mtime-busted so /api/integrations/reload picks up in-place edits. In prod the URL is *not* cache-busted — Node's ESM loader returns the cached module without leaking memory, so updates to an already-loaded integration need `pnpm openmapx services restart app-api`. The install/update job logs surface that hint automatically. New installs work without restart. Replaces the per-route Fastify registration with a single mini-dispatcher + sorted route table so reload genuinely replaces the route map. Pattern subset documented inline: literal segments, `:param`, and a trailing `*`. Tests + typecheck Updates packages/cli/__tests__/cli-integrations.test.ts to match the new contract (asserts bare-spec externals in the bundle, rejects artifacts shipping node_modules/, drops the deprecated-deps tests). All passing: packages/cli 129, packages/core 284, apps/api 1069 (+4 skipped).
1 parent 2d54818 commit cff4fe0

33 files changed

Lines changed: 2105 additions & 322 deletions

.dockerignore

Lines changed: 28 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,13 +1,40 @@
11
**/node_modules
22
**/.next
33
**/.turbo
4+
.pnpm-store
5+
.pnp
6+
.pnp.*
47
.git
58
.gitignore
9+
.claude
10+
.gemini
11+
.husky
612
*.md
713
LICENSE
814
.env*
915
!.env.example
1016
infra/docker/data
17+
infra/docker/docker-compose.generated.yml
18+
infra/docker/docker-compose.generated.hardlinks.json
19+
infra/docker/service-selection.json
20+
infra/docker/services/transitous/api-keys.json
21+
services/motis/tools/transitous/api-keys.json
22+
services/.community
23+
custom_integrations
24+
integrations/*/config.json
1125
**/.DS_Store
12-
**/tsconfig.tsbuildinfo
26+
**/coverage
27+
apps/api/dist
28+
services/data-manager/dist
29+
packages/*/dist
30+
apps/web/next-env.d.ts
31+
apps/web/public/sw.js
32+
apps/web/public/icons/maki
33+
apps/web/public/icons/temaki
34+
**/*.tsbuildinfo
35+
pnpm-debug.log*
36+
npm-debug.log*
37+
yarn-debug.log*
38+
yarn-error.log*
39+
*.pem
1340
docs/

.gitignore

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -6,15 +6,18 @@
66

77
# Dependencies
88
node_modules
9+
.pnpm-store/
910
.pnp
1011
.pnp.js
1112

1213
# Local env files
1314
.env
15+
.env.*
1416
.env.local
1517
.env.development.local
1618
.env.test.local
1719
.env.production.local
20+
!.env.example
1821

1922
# Testing
2023
coverage
@@ -35,10 +38,14 @@ dist
3538
npm-debug.log*
3639
yarn-debug.log*
3740
yarn-error.log*
41+
pnpm-debug.log*
3842

3943
# Playwright
4044
.playwright-mcp/
4145

46+
# TypeScript
47+
*.tsbuildinfo
48+
4249
# Misc
4350
.DS_Store
4451
*.pem
@@ -54,6 +61,9 @@ infra/docker/data/
5461
# Community integrations (user-installed, not version-controlled)
5562
custom_integrations/
5663

64+
# Generated runtime modules served to community integrations via import maps
65+
apps/web/public/runtime/
66+
5767
# Integration local config (may contain API keys)
5868
integrations/*/config.json
5969

@@ -67,4 +77,5 @@ infra/docker/docker-compose.generated.yml
6777
infra/docker/docker-compose.generated.hardlinks.json
6878
infra/docker/service-selection.json
6979
infra/docker/services/transitous/api-keys.json
80+
services/motis/tools/transitous/api-keys.json
7081
services/.community/

.npmrc

Lines changed: 0 additions & 6 deletions
This file was deleted.

README.md

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -8,7 +8,7 @@
88
[![CodeQL](https://github.com/Medformatik/openmapx/actions/workflows/codeql.yml/badge.svg)](https://github.com/Medformatik/openmapx/actions/workflows/codeql.yml)
99
[![Docker](https://github.com/Medformatik/openmapx/actions/workflows/docker.yml/badge.svg)](https://github.com/Medformatik/openmapx/actions/workflows/docker.yml)
1010
[![Node.js](https://img.shields.io/badge/node-%3E%3D24-43853d?logo=node.js&logoColor=white)](https://nodejs.org)
11-
[![pnpm](https://img.shields.io/badge/pnpm-10-f69220?logo=pnpm&logoColor=white)](https://pnpm.io)
11+
[![pnpm](https://img.shields.io/badge/pnpm-11-f69220?logo=pnpm&logoColor=white)](https://pnpm.io)
1212
[![TypeScript](https://img.shields.io/badge/TypeScript-5.9-3178c6?logo=typescript&logoColor=white)](https://www.typescriptlang.org)
1313
[![Next.js](https://img.shields.io/badge/Next.js-16-000?logo=nextdotjs&logoColor=white)](https://nextjs.org)
1414
[![Fastify](https://img.shields.io/badge/Fastify-5-000?logo=fastify&logoColor=white)](https://fastify.dev)
@@ -120,7 +120,7 @@ pnpm openmapx services enable|disable|list|start|stop|restart|build|status|logs
120120
pnpm openmapx compose render|up|down
121121
pnpm openmapx data download|link|status
122122
pnpm openmapx repos list|add|refresh|remove # community service repositories
123-
pnpm openmapx integrations list|install|validate|build
123+
pnpm openmapx integrations list|install|validate|build|package
124124
pnpm openmapx users list|create|promote
125125
pnpm openmapx check # environment + manifest validation
126126
```
@@ -151,7 +151,7 @@ See [Admin Panel](https://github.com/Medformatik/openmapx/wiki/Admin-Panel) for
151151
| Routing & transit | Valhalla, OSRM, MOTIS, OpenTripPlanner |
152152
| Geocoding | Photon, Nominatim, Pelias |
153153
| Tiles | TileServer GL, Martin (PostGIS vector tiles) |
154-
| Tooling | Turborepo, pnpm 10, Biome, Vitest, Husky + Commitlint, Changesets, Docker Compose v2, Traefik |
154+
| Tooling | Turborepo, pnpm 11, Biome, Vitest, Husky + Commitlint, Changesets, Docker Compose v2, Traefik |
155155
| Language | TypeScript end-to-end (Node 24+) |
156156

157157
## Contributing

apps/api/Dockerfile

Lines changed: 7 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -7,7 +7,7 @@ RUN corepack enable && corepack prepare $(node -e "console.log(require('/tmp/pac
77
# Install all dependencies (for building)
88
FROM base AS deps
99
WORKDIR /app
10-
COPY package.json pnpm-lock.yaml pnpm-workspace.yaml .npmrc ./
10+
COPY package.json pnpm-lock.yaml pnpm-workspace.yaml ./
1111
COPY packages/core/package.json packages/core/
1212
COPY packages/shared-mobility/package.json packages/shared-mobility/
1313
COPY packages/mobility-formats/package.json packages/mobility-formats/
@@ -21,7 +21,7 @@ RUN pnpm install --frozen-lockfile
2121
# Install production dependencies only (for runtime)
2222
FROM base AS prod-deps
2323
WORKDIR /app
24-
COPY package.json pnpm-lock.yaml pnpm-workspace.yaml .npmrc ./
24+
COPY package.json pnpm-lock.yaml pnpm-workspace.yaml ./
2525
COPY packages/core/package.json packages/core/
2626
COPY packages/shared-mobility/package.json packages/shared-mobility/
2727
COPY packages/mobility-formats/package.json packages/mobility-formats/
@@ -63,14 +63,18 @@ WORKDIR /app
6363
# `curl -fs http://localhost:3001/health`). Alpine ships wget but not curl,
6464
# so we install it explicitly. ca-certificates gives node outgoing HTTPS.
6565
#
66+
# git is required by admin Store / community-service repository installs:
67+
# both the community integration installer and service-repo refresh path clone
68+
# or fetch allowlisted Git URLs from inside the app-api container.
69+
#
6670
# docker-cli + docker-cli-compose enable the admin GUI's service-lifecycle
6771
# and data-maintenance workflows: admin-ops.ts and admin-cli.ts shell out to
6872
# `docker compose -f <generated>.yml up/stop/restart` and to `openmapx ...`.
6973
# The socket itself is mounted via services/app-api/service.json's
7074
# `@docker-socket` bindMount. Without the CLI in the image those handlers
7175
# would fail even when the socket is available. Kept behind the same apk
7276
# install for a single image layer.
73-
RUN apk add --no-cache curl ca-certificates docker-cli docker-cli-compose
77+
RUN apk add --no-cache curl ca-certificates git docker-cli docker-cli-compose
7478

7579
ENV NODE_ENV=production
7680
ENV PORT=3001

apps/api/src/db/installed-integration-schema.ts

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -7,7 +7,7 @@ export const installedIntegration = pgTable(
77
id: text("id").primaryKey(),
88
repository: text("repository").notNull(),
99
installedVersion: text("installed_version").notNull(),
10-
sourceType: text("source_type").notNull().default("registry"), // registry | git | local
10+
sourceType: text("source_type").notNull().default("registry"), // registry | artifact
1111
installedAt: timestamp("installed_at").defaultNow().notNull(),
1212
updatedAt: timestamp("updated_at").defaultNow().notNull(),
1313
installedBy: text("installed_by").references(() => user.id, { onDelete: "set null" }),

0 commit comments

Comments
 (0)