Skip to content

local-development.md

Latest commit

 

History

History
649 lines (441 loc) · 48.3 KB

local-development.md

File metadata and controls

649 lines (441 loc) · 48.3 KB
Raw

Local Development

Overview

ERNIE uses a Docker-first local workflow.

  • Fast Mode is the default path for day-to-day development.
  • Optional profiles are available for assessment-specific and parity-specific work.
  • Canonical validation entry points remain npm run check:backend, npm run check:frontend, and npm run check:parity.

Host-side frontend commands require local node_modules in the repository checkout. Run npm ci after cloning and whenever package-lock.json changes. Use npm install only when intentionally adding or updating dependencies so npm can update the lockfile.

Mode Purpose Command
Fast Mode Start the core development stack only npm run docker:dev:up
Assessment profile Start F-UJI and the dedicated persistent assessment workers; also set FUJI_ENABLED=true in .env.docker npm run docker:dev:assessment
Parity profile Start the parity stack including F-UJI and the dedicated persistent assessment workers; also set FUJI_ENABLED=true in .env.docker npm run docker:dev:parity

Fast Mode is the default because it keeps the profile-gated F-UJI service out of the normal startup path.

Windows Recommendation

Preferred: WSL2 checkout

WSL2 is the recommended Windows setup because Docker bind mounts and host-side Node tooling are significantly faster inside the WSL filesystem.

  1. Install Docker Desktop with WSL2 integration enabled.
  2. Clone the repository inside your WSL home directory, for example ~/src/ernie.
  3. Open the project through VS Code Remote - WSL.
  4. Run Docker Compose and host-side Node commands from the WSL shell.
  5. Use your Windows browser for https://ernie.localhost:3333 if preferred.

The repository recommends Microsoft's TypeScript 7 extension for VS Code and enables its native language server with the workspace-local TypeScript 7 package. Accept the workspace extension recommendation after opening the checkout. Run npm ci first so the configured SDK path is available.

Supported fallback: Windows checkout on NTFS

If the repository stays under D:\ or another NTFS path:

  • expect slower bind-mount performance than WSL2
  • keep VITE_USE_POLLING=true enabled
  • use the public/hot troubleshooting step below if HMR becomes unreliable

Quick Start

  1. Generate certificates.

    Windows PowerShell:

    .\docker\generate-certs.ps1

    WSL, Git Bash, or another POSIX shell:

    ./docker/generate-certs.sh

  2. Create the Docker environment file.

    Windows PowerShell:

    Copy-Item .env.docker.example .env.docker

    WSL, Git Bash, or another POSIX shell:

    cp .env.docker.example .env.docker

  3. Install host-side Node dependencies for frontend validation.

    npm ci

    This installs the local node_modules required by ESLint, TypeScript, Vitest, OpenAPI linting, and Playwright.

  4. Start Fast Mode.

    npm run docker:dev:up

  5. Trust docker\traefik\certs\localhost.crt on Windows if your browser warns about the local TLS certificate.

  6. Open the application.

    • Main URL: https://ernie.localhost:3333
    • Localhost fallback after switching ERNIE_DEV_HOST and ERNIE_DEV_SESSION_DOMAIN: https://localhost:3333

    If ernie.localhost does not resolve, add 127.0.0.1 ernie.localhost to your hosts file.

  7. Create the first administrator account.

    npm run artisan -- add-user "Admin Name" admin@example.com SecurePassword

The Docker entrypoints install missing Composer dependencies and container-local npm dependencies, run migrations, and seed baseline data when the database is empty. Host-side frontend commands still require the local npm ci step above.

For day-to-day Laravel commands, use the npm wrappers. They run inside the app container, and generated files are written into the bind-mounted repository, so they still appear in your host checkout:

npm run artisan -- make:controller TestController

Profiles And Services

Default Fast Mode services:

  • Traefik
  • app
  • webserver
  • vite
  • db
  • redis
  • queue
  • scheduler

The scheduler waits for the application and database to become healthy, refreshes the persisted license usage ranking once at startup, and then runs Laravel's configured schedule continuously. The ranking is refreshed weekly after that initial backfill, so opening the Data Editor never needs to aggregate all stored resource-license associations.

Optional profiles:

  • assessment starts F-UJI plus the dedicated assessment-queue workers; set FUJI_ENABLED=true in .env.docker when the app should use it
  • parity starts the same assessment services together with the parity profile; set FUJI_ENABLED=true in .env.docker when the app should use it

Common startup commands:

npm run docker:dev:up
npm run docker:dev:assessment
npm run docker:dev:parity

Persistent FAIR assessment queue

Resource and IGSN assessments use database-backed runs and one short queue job per resource. Closing the browser, restarting a worker, or deploying the application therefore does not discard the run snapshot or its progress. Opening /assessment again reconnects to an active run; pressing Check while a run is active returns that same run, and a paused run is resumed instead of replaced.

The assessment profile starts two assessment-queue workers by default. Their shared Redis limiter permits at most 80 F-UJI requests per rolling 60-second window and spaces request starts by at least 750 ms. These defaults leave headroom below the local F-UJI limit of 100 requests per minute. To diagnose pressure, reduce FUJI_ASSESSMENT_CONCURRENCY to 1 before increasing any request limits.

The relevant settings are documented in .env.docker.example. Keep these relationships intact when changing them:

  • FUJI_ASSESSMENT_ITEM_TIMEOUT must remain higher than FUJI_TIMEOUT.
  • FUJI_ASSESSMENT_LEASE_SECONDS and FUJI_ASSESSMENT_QUEUE_RETRY_AFTER must remain higher than the item timeout.
  • FUJI_ASSESSMENT_QUEUE_CONNECTION must use a persistent driver; sync and null are rejected.
  • Every web and assessment-worker process must use the same Redis cache so request limiting and start locks are shared.

For a one-worker local load comparison, run:

docker compose --env-file .env.docker -f docker-compose.dev.yml --profile assessment up --build --scale assessment-queue=1

Command Reference

Task Recommended place Command
Start the core stack Host shell npm run docker:dev:up
Install host-side frontend dependencies Host shell npm ci
Start the backend services needed for PHP checks Host shell npm run docker:dev:backend:d
Stop the stack Host shell npm run docker:dev:down
Reset Docker volumes Host shell npm run docker:dev:reset
Laravel Artisan npm wrapper into app container npm run artisan -- <command>
Example controller generator npm wrapper into app container npm run artisan -- make:controller TestController
Refresh the license usage ranking npm wrapper into app container npm run artisan -- rights:update-usage-count
Composer npm wrapper into app container npm run composer:app -- <command>
Pest (2 GB, optimized complete suite) Host shell via npm wrapper npm run test:php
Pest deprecation details Host shell via npm wrapper npm run test:php:deprecations
MySQL-sensitive Pest slice Host shell via npm wrapper npm run test:php:mysql-sensitive
PHPStan Host shell via npm wrapper npm run phpstan:check
Vitest Host shell npm run test:run
Vitest performance diagnosis Host shell npm run test:doctor
ESLint check Host shell npm run lint:check
ESLint auto-fix Host shell npm run lint
TypeScript Host shell npm run types
TypeScript application watcher Host shell npm run types:watch
TypeScript test watcher Host shell npm run types:watch:test
Playwright against the dev stack Host shell npm run test:e2e:devstack
Canonical backend validation Host shell npm run check:backend
Canonical frontend validation Host shell npm run check:frontend
Canonical parity validation Host shell npm run check:parity

Environment Files

  • .env.docker is the Docker-oriented local environment file.
  • .env is the Laravel application environment used inside the containers.
  • The development entrypoint copies .env.docker to .env when .env does not already exist.
  • The npm Docker wrappers always pass --env-file .env.docker so Compose and Laravel use the same source of truth.
  • Docker-managed node_modules live in the named Docker volume, not in your host checkout.
  • Complete Pest runs copy the current checkout into the Docker-managed ernie-pest-workspace volume before execution. This avoids repeated Windows/macOS bind-mount reads while leaving the development source mount and focused test workflow unchanged.

MySQL version and existing local data

The local database service is pinned to MySQL 9.7, and its healthcheck verifies the 9.7.x server series. MySQL-backed tests use this same service. Do not downgrade the service to MySQL 8 when an existing data directory fails to start.

The Compose volume is named ernie-db-data-mysql-9-7 so an older ernie-db-data volume cannot accidentally be mounted into MySQL 9.7. This is necessary because MySQL 9.7 rejects a direct in-place upgrade from a data directory created by a non-LTS MySQL 8.0 release. Compose creates and seeds the new 9.7 volume on first startup; the old volume is retained and can be migrated separately with a logical dump and restore if its local data is still needed.

Dockerfile and Dockerfile.dev also contain a MySQL 8.4 build stage. That stage contributes only the separately named mysql-legacy-mysqldump client needed to export the external MySQL 5.6 IGSN database. It never runs as ERNIE's database server.

Log retention and downloads

The administration log viewer can download the application log for the last 24 hours, 7 days, or 30 days. It reads both storage/logs/laravel.log and Laravel's daily files named laravel-YYYY-MM-DD.log. A download can only contain history that is still present on disk. Deployments using the daily log channel must therefore keep LOG_DAILY_DAYS at 31 or more for complete 30-day downloads. The production template selects that channel with LOG_STACK=daily; the development templates continue to use single. The repository environment templates and Laravel's default use 31 days, while an environment-specific override takes precedence.

Host VM resource monitoring

The administrator-only /logs page includes CPU and memory history for the entire host VM. Production and Stage enable the collector in their Compose files. Only the scheduler container receives read-only bind mounts for the host's /proc/stat and /proc/meminfo; the web application reads normalized samples from MySQL and has no host filesystem access.

The scheduler records one sample per minute. The page displays five-minute averages for the last 24 hours or 30-minute averages for the last 7 days. Raw samples are retained for 30 days by default and pruned daily. The first sample after enabling monitoring or rebooting the VM establishes a CPU baseline, so a CPU percentage appears after the next consecutive sample. Missing intervals remain visible as gaps instead of being filled with zeroes.

Local collection defaults to SYSTEM_METRICS_ENABLED=false. Docker Desktop would report the resource usage of its internal Linux VM rather than the physical Windows or macOS computer, so /logs shows an intentional disabled state locally. Parser and collector tests use deterministic fixtures and never depend on the development host's /proc.

Useful Stage or Production checks are:

docker compose -f docker-compose.stage.yml exec scheduler test -r /host/proc/stat
docker compose -f docker-compose.stage.yml exec scheduler test -r /host/proc/meminfo
docker compose -f docker-compose.stage.yml exec scheduler php artisan system-metrics:collect

No Docker socket, privileged container, host PID namespace, or writable host mount is required. Override SYSTEM_METRICS_RETENTION_DAYS only with a positive number. Disabling collection preserves existing history until normal retention removes it.

Anonymous public traffic history

The administrator-only /logs page also shows the typical number of unique signed-out visitors by weekday and hour for published landing pages and the DOI and IGSN portals. The application deduplicates visitors for the current UTC hour in the shared cache using a bucket-specific HMAC of IP address and user agent. This HMAC identifier exists only in short-lived cache key names and expires shortly after its UTC hour. Only hourly counters are persisted; no IP address, user agent, cookie, session identifier, or visitor hash is stored in MySQL or application logs. Authenticated users, empty user agents, and recognizable crawlers are excluded. Extend the built-in heuristic list with a comma-separated BOT_PROTECTION_ADDITIONAL_CRAWLER_USER_AGENTS value when required.

PUBLIC_TRAFFIC_ENABLED defaults to false locally. Stage and Production enable it in Compose. Their scheduler runs public-traffic:observe-availability every minute against the configured public /health URL and verifies the shared cache before marking the minute as observed. Only completed hours with all 60 minutes observed contribute to the heatmap. This keeps a healthy zero-visitor hour distinct from an application or analytics outage. Aggregates are retained for 400 days by default and pruned daily.

For a controlled local check, enable the feature and point the probe at a trusted reachable health URL before running:

docker compose --env-file .env.docker -f docker-compose.dev.yml exec app php artisan public-traffic:observe-availability
docker compose --env-file .env.docker -f docker-compose.dev.yml exec app php artisan public-traffic:prune

The UI groups UTC buckets in Europe/Berlin, including daylight-saving time, and offers rolling 4-, 12-, and 52-week views. Recommendations remain in the collecting state until every weekday/hour cell has at least one complete observation.

DataCite import mode

Keep DATACITE_TEST_MODE=true for local development and Stage. Eligible imported resources and every newly imported IGSN receive their local landing page, but the import never writes metadata to either DataCite API in this mode.

GEOFON Seismic Events keep the external landing-page target supplied by DataCite when it points to a GEOFON host. Legacy http:// GEOFON URLs are stored as https://, and retired /db/eqpage.php?id=... event targets are stored directly under the canonical https://geofon.gfz.de/eqinfo/event.php?id=... URL. Findable records receive a published external landing page, so their ERNIE workflow status is published; this applies based on the selected or assigned datacenter and is not limited to one DOI prefix.

With DATACITE_TEST_MODE=false on Production, the same local landing pages are created and the newly imported, published records enter a separate DataCite synchronization phase. That phase exports the complete ERNIE metadata and changes the DOI target URL to the new landing page. Failed updates do not roll back the import or landing page and can be retried from the completed import dialog.

Production uses separate DataCite Repository accounts for ordinary GFZ DOIs and legacy IGSNs. Configure DATACITE_USERNAME / DATACITE_PASSWORD and DATACITE_CLIENT_ID for the ordinary DOI repository and DATACITE_IGSN_USERNAME / DATACITE_IGSN_PASSWORD for the GFZ.IGSN repository that owns prefix 10.60510. ERNIE selects the IGSN credentials automatically for that prefix; using the ordinary DOI credentials results in a DataCite HTTP 403 response. Audits against the test API additionally require DATACITE_TEST_CLIENT_ID.

GEOFON seismic-event landing-page repair

Use the authenticated, dry-run-first repair command for existing GEOFON Seismic Events Resources whose external landing page still uses the retired db/eqpage.php route. It checks the local and DataCite URLs independently, requires the event ID in both URLs to match the DOI suffix, and accepts only the known GEOFON event DOI namespaces and hosts. It also reports a matching retired local URL assigned to any other datacenter as manual review and never updates that Resource.

Start with a complete audit:

npm run artisan -- resources:repair-geofon-event-landing-page-urls \
    --report=storage/app/geofon-event-url-audit.csv

The dry run performs authenticated DataCite GETs and bounded reachability checks against the canonical https://geofon.gfz.de/eqinfo/event.php?id=... target, but it sends no PUT and changes no local row. Review every wrong-datacenter, unknown URL, event-ID mismatch, unreachable target, and API error before applying anything.

Apply and verify one representative DOI first:

npm run artisan -- resources:repair-geofon-event-landing-page-urls \
    --apply --force-production \
    --doi=10.1594/gfz.geofon.gfz2011axdw \
    --report=storage/app/geofon-event-url-pilot.csv

For each required DataCite update, ERNIE saves the complete source record below storage/app/private/geofon-event-url-updates/, sends only the new url, and confirms it with another authenticated GET. The local domain and path are updated transactionally only after DataCite is confirmed; an independently stale local URL is still repairable when DataCite is already current. Authentication failures stop subsequent records, while per-record validation, reachability, verification, and concurrency failures remain visible in the CSV without overwriting local data.

After the pilot resolves directly to the canonical GEOFON page, run the complete apply and then repeat the dry run:

npm run artisan -- resources:repair-geofon-event-landing-page-urls \
    --apply --force-production \
    --report=storage/app/geofon-event-url-applied.csv

npm run artisan -- resources:repair-geofon-event-landing-page-urls \
    --report=storage/app/geofon-event-url-verification.csv

The final dry run must report every eligible Resource as already_current. Use repeatable --doi filters for targeted retries, --after-id=<resource-id> to resume after the last processed Resource, and --limit=<count> for bounded batches. Production writes always require both --apply and --force-production; test mode never reaches the production API.

DataCite Kernel 4 upgrade for imported resources

Every metadata update for an existing DOI now asks DataCite to store the record with Kernel 4. In the Data Editor, any non-empty DOI therefore produces the Update Metadata action regardless of the local draft, curation, review, or published status. This is an update of the existing DOI, not another registration.

Use the authenticated, dry-run-first upgrade command to normalize already imported repository records. It lists every state visible to the configured Repository account, intersects the result with local ERNIE resources by normalized DOI, and excludes IGSNs and unconfigured prefixes. No database migration is involved.

Start with an audit and inspect all manual_review, not_imported, and error rows:

npm run artisan -- resources:upgrade-datacite-schema \
    --report=storage/app/datacite-schema-upgrade-dry-run.csv

The dry run never sends a PUT. Kernel 3, Kernel 2.x, and records without a schema version are automatically eligible only when types.resourceTypeGeneral is valid for Kernel 4 and no legacy contributorType: Funder remains. Unknown schema versions and incompatible metadata stay unchanged for manual review.

Apply one representative DOI from each legacy group first:

npm run artisan -- resources:upgrade-datacite-schema \
    --apply --force-production \
    --doi=10.5880/example \
    --report=storage/app/datacite-schema-upgrade-pilot.csv

--force-production is mandatory only when --apply targets the production API. It is an explicit safety acknowledgement; the command remains non-interactive and still performs every per-record preflight. Confirm that the pilot retained its DOI state, landing-page URL, and resource type before running the complete upgrade:

npm run artisan -- resources:upgrade-datacite-schema \
    --apply --force-production \
    --report=storage/app/datacite-schema-upgrade-applied.csv

Apply runs send only doi, schemaVersion, and the existing types object. Before each PUT, the complete DataCite source record is saved below storage/app/private/datacite-schema-upgrades/; this location is not publicly served. API or verification failures are isolated in the CSV, except that HTTP 401 or 403 stops subsequent writes. Resume a bounded run with --after-id=<resource-id> and --limit=<count>, or retry individual records with repeatable --doi options.

After applying, repeat the dry run. Successfully upgraded records appear as already_current and do not cause another write, making the process idempotent. Archive the apply report and private snapshots according to the operating retention policy.

Legacy line coverage enrichment

Newly created resources from DataCite are optionally enriched with profile lines from non-empty sumario-pmd.coverage.wkt values. ERNIE stores each valid coordinate chain as geo_type = line with its ordered points in polygon_points, so the Data Editor and landing pages retain the original line geometry. On later DataCite exports, the existing thin-polygon workaround is used because DataCite does not provide a line geometry type.

The enrichment replaces a DataCite bounding box only when all four legacy bounds match and exactly one safe candidate can be identified, using the place description to resolve equal boxes where possible. Other GeoLocations are preserved. Invalid geometry, incomplete bounds, ambiguous matches, and legacy database failures keep the imported DataCite metadata and do not fail the import.

This enrichment runs only while creating a new DataCite resource. Duplicate, skipped, and repair paths do not add lines to resources that already exist in ERNIE; there is no automatic backfill.

Legacy description break cleanup

New SUMARIO imports correct the duplicated paragraph breaks produced by the legacy XML export before storing descriptions. The correction is pairwise: two consecutive <br> tags become one, three become two, four become two, and in general a run of n break tags or plain-text newline tokens becomes ceil(n / 2). Whitespace between tags and the variants <br>, <br/>, and <br /> are supported; unrelated text and HTML remain unchanged.

The migration adds resources.legacy_description_breaks_normalized_at as a durable one-time marker because applying the pairwise rule twice would remove intentional spacing. The cleanup considers both resources marked with legacy_source = sumario-pmd and older unmarked resources whose normalized DOI has exactly one match in SUMARIO. Ambiguous DOI matches are reported for manual review. The configured metaworks connection must therefore be reachable for every run, although the command never writes to the legacy database.

Deploy the migration, take the normal ERNIE database backup, and audit the complete selection before applying changes:

npm run artisan -- migrate --force
npm run artisan -- resources:repair-legacy-description-breaks \
    --report=storage/app/legacy-description-breaks-dry-run.csv
npm run artisan -- resources:repair-legacy-description-breaks \
    --apply --after-id=0 --limit=500 --chunk=100 \
    --report=storage/app/legacy-description-breaks-applied.csv

Use repeatable --doi or --legacy-id options for targeted audits. --after-id refers to the ERNIE resources.id; the command always prints the last scanned ID, including batches containing only non-legacy candidates whose CSV has no data rows. Review the CSV and use that printed ID before continuing with another bounded batch. Apply runs update all descriptions of one resource transactionally, reject concurrent edits, invalidate a changed published landing-page cache, and never process an already marked resource again.

With DATACITE_TEST_MODE=false, every changed resource with a DOI is queued for a complete metadata synchronization through the imports queue. In test mode the local repair still applies but no DataCite request is made. Sync failures do not roll back local changes; retry them with the run UUID printed by the apply command:

npm run artisan -- resources:repair-legacy-description-breaks --retry-sync=<sync-run-uuid>

Legacy temporal coverage backfill

The temporal-coverage migration adds nullable columns to geo_locations; it does not guess or manufacture values for existing rows. After deploying the migration, the original sumario-pmd.coverage.start and coverage.end values can be copied into already imported ERNIE resources with the dry-run-first backfill command.

By default, the command considers only resources with the exact legacy_source = sumario-pmd and legacy_source_id recorded by the SUMARIO import. It matches each legacy coverage to an existing GeoLocation by its spatial coordinates, then uses a normalized description or the original one-to-one position only where that is unambiguous. A legacy coverage without spatial identity is added as a temporal/place-only GeoLocation. Existing equal values are left unchanged. If any temporal field conflicts, the matched GeoLocation remains completely unchanged and is reported for manual review; missing fields are filled only when the complete merge is conflict-free.

Run the migration and audit before applying changes:

npm run artisan -- migrate --force
npm run artisan -- resources:backfill-legacy-temporal-coverages --report=storage/app/legacy-temporal-coverage-dry-run.csv

Review every manual_review, missing_legacy, and error row in the CSV. Then apply the safe rows in bounded batches:

npm run artisan -- resources:backfill-legacy-temporal-coverages --apply --after-id=0 --limit=500 --chunk=100 --report=storage/app/legacy-temporal-coverage-applied.csv

Use repeatable --doi or --legacy-id options for a targeted rollout. --after-id always refers to the ERNIE resources.id shown in the report. The command is idempotent and invalidates the rendered cache of a changed published landing page.

Imports created before ERNIE recorded legacy_source_id require an explicit DOI fallback. Audit a small selection first because this mode also inspects otherwise unlinked ERNIE resources whose DOI exists in SUMARIO:

npm run artisan -- resources:backfill-legacy-temporal-coverages --match-by-doi --doi=10.5880/example --report=storage/app/legacy-temporal-coverage-doi-audit.csv
npm run artisan -- resources:backfill-legacy-temporal-coverages --apply --match-by-doi --doi=10.5880/example

The application and queue containers need working access to the configured metaworks connection while the command runs. The backfill reads but never modifies the legacy database. Take the normal ERNIE database backup before the apply run; a nonzero exit code indicates processing errors, while manual-review rows deliberately remain unchanged and do not fail the complete run.

Exact subject duplicate cleanup

Metadata imports classify each DataCite subject only once. Resources imported before that fix can be audited with a dry-run-first command. By default it considers controlled subjects only and treats rows as duplicates only when value, language, scheme, scheme URI, value URI, classification code, and breadcrumb path are all exactly equal. The smallest Subject ID survives.

Audit GEMET and MSL first and retain the CSV for review:

npm run artisan -- subjects:deduplicate \
    --scheme="GEMET - GEneral Multilingual Environmental Thesaurus" \
    --scheme="EPOS MSL vocabulary" \
    --report=storage/app/subject-duplicates-dry-run.csv

After taking the normal database backup and reviewing the report, apply the same bounded selection:

npm run artisan -- subjects:deduplicate --apply \
    --scheme="GEMET - GEneral Multilingual Environmental Thesaurus" \
    --scheme="EPOS MSL vocabulary" \
    --after-resource-id=0 --limit=500 --chunk=100 \
    --report=storage/app/subject-duplicates-applied.csv

The repeatable --doi option narrows an audit further. Use --include-free only when exact free-keyword duplicates should also be considered. Apply runs are transactional per resource and idempotent; they remove stale Subject assistance rows and invalidate affected keyword and landing-page caches. A nonzero exit code indicates processing errors.

Testing Data Editor registration safely

Keep DATACITE_TEST_MODE=true and use test Repository credentials when exercising the Data Editor's Register or Update Metadata actions locally. Validate, Save Draft, autosave, Preview LP, and Show LP are local-only actions and must not produce a DataCite request. The two DataCite write actions run complete client validation, show an explicit confirmation before their action-specific save, and automatically continue through landing-page setup when a page is missing.

After a new test DOI is registered, the success dialog displays the number of locally published Resources plus published IGSNs. This is deliberately smaller than or equal to the combined sidebar badges: a record counts only when it has a non-empty DOI and a published landing page. Draft, curation, and review records are excluded. The dialog redirects to /resources after five seconds; operating-system reduced-motion settings suppress confetti without changing the result or countdown.

There is no separate post-import sync flag: DATACITE_TEST_MODE is the only switch. The queue worker must consume the imports queue so the bounded synchronization jobs can complete.

Testing queued IGSN batch registration safely

The IGSN list can queue up to 1000 selected samples for DataCite registration or metadata updates. Keep DATACITE_TEST_MODE=true, configure valid DataCite Test credentials, and use only disposable identifiers when testing this workflow locally. The start request returns 202 Accepted; it never performs the DataCite writes in the web request.

Each run and its ordered items are stored in igsn_registration_runs and igsn_registration_items. One short-lived job processes one item and dispatches the next job on the queue configured by DATACITE_QUEUE, which defaults to datacite. Both the app and queue services must use a persistent queue connection such as database; sync and null are deliberately rejected. Every Docker environment forwards DATACITE_QUEUE to both services, and its worker consumes the configured queue.

The effective DataCite test/production mode and endpoint are snapshotted when a run starts. Credentials are never stored with the run and are read from the current server configuration by each job. A changed mode or endpoint pauses the run before another write. Beginner runs remain on DataCite Test even when a worker executes without a signed-in browser user.

Closing the progress dialog, navigating away, restarting a worker, or reloading the browser does not cancel a run. Return to /igsns and use View registration progress to inspect it. Cancellation takes effect between external requests. Failed items can be retried without resending successful items; a resumed item also checks whether DataCite already accepted an earlier create request before it attempts another create.

For a safe local check:

  1. Start the Docker stack and confirm the queue service consumes datacite.
  2. Create a small set of synthetic IGSNs with published landing pages.
  3. Confirm the progress dialog says DataCite Test, then use Register Selected.
  4. Close and reopen the dialog while the run is active, and verify the same run and counters return.
  5. Exercise cancellation or retry only with disposable test identifiers. Never use production credentials for automated or local development tests.

Legacy IGSN enrichment

Single-IGSN imports preload their complete DataCite family and the corresponding legacy DIF metadata before writing resources. The public legacy IGSN portal is the mandatory source for this strict preflight. If the portal is unreachable, returns invalid JSON, or contains malformed DIF data, the complete single import fails without creating any new resource, IGSN metadata, relationship, datacenter assignment, or landing page. Existing ERNIE resources remain unchanged and are not backfilled.

The default portal endpoint and retry settings are defined by:

  • GFZ_IGSN_PORTAL_PROXY_URL
  • GFZ_IGSN_PORTAL_CONNECT_TIMEOUT
  • GFZ_IGSN_PORTAL_TIMEOUT
  • GFZ_IGSN_PORTAL_RETRY_TIMES
  • GFZ_IGSN_PORTAL_RETRY_SLEEP_MS
  • GFZ_IGSN_PORTAL_RETRY_JITTER_MS

Keep the portal URL on HTTPS. Retries include the observed failure mode in which the proxy answers with HTTP 200 but the body is not valid JSON. The queue job records a stable error_code such as legacy_source_unavailable or legacy_invalid_payload in the import progress instead of silently creating a DataCite-only partial record.

Authenticated Solr and the direct legacy database remain optional enrichment sources for non-single imports. Enable IGSN_LEGACY_DB_ENABLED only after the configured TLS and credentials have been verified from both the app and queue containers. Network reachability of port 3306 alone does not prove that the legacy database connection works.

Both app and queue services must use QUEUE_CONNECTION=database, and the worker command must consume imports. The single-import start endpoint returns 202 Accepted; later portal or persistence failures are reported through the import status endpoint.

Backfill complete legacy IGSN DIF metadata

New legacy imports inspect every non-empty DIF 1.1, 1.2, and 1.3 leaf across all sample blocks and project the approved metadata into typed ERNIE columns and relations. This includes root hasDocument publication DOIs as DataCite Cites, funders, Available and Collected dates, contributors, geological ages, Rock Type, request information, structured methods, drilling lengths, launch/navigation values, and elevation ranges. ERNIE does not retain a raw DIF payload: unknown non-empty paths are reported with their complete namespace-independent path and sample index so that they require an explicit mapping decision. The known supplemental IsCitedBy copies are intentionally ignored because their direction is wrong and the equivalent root publications are authoritative. Total Length remains local IGSN drilling metadata and is never exported as DataCite sizes.

Use the dry-run-first command to audit and repair IGSNs already stored in ERNIE across all configured legacy datacenters:

npm run artisan -- igsn:backfill-legacy-dif-metadata --report=storage/app/igsn-dif-dry-run.csv
npm run artisan -- igsn:backfill-legacy-dif-metadata --datacenter=IGSNDB.ICDP --doi=ICDP5052ECZI101
npm run artisan -- igsn:backfill-legacy-dif-metadata --apply --report=storage/app/igsn-dif-applied.csv

Review missing DIF records, unknown non-empty paths, scalar and privacy conflicts, image availability, and technical errors before using --apply. Apply mode is additive: it fills missing values and appends missing repeated values while preserving conflicting local values for manual review. The same command validates allowlisted external ICDP sample images. A definitive 404/410 or invalid image response is reported as unavailable and removes only the public sample_image_external_url; the validated source descriptor remains stored for auditing. A timeout, transport failure, rate limit, or temporary 5xx response is reported as an image probe error and does not remove a previously published URL. Repeat --doi or --datacenter for a restricted scope, use --limit for a bounded batch, and resume after the printed Resource ID with --after-id. Portal requests are limited to 100 handles per request even when a larger --chunk is supplied.

The console summary reports unavailable images and temporary probe errors separately. CSV reports include sample_image_status, sample_image_url, and sample_image_message. Image-only changes use the same transaction, landing-page cache invalidation, DataCite synchronization, CSV audit, and --retry-sync path as every other field handled by igsn:backfill-legacy-dif-metadata; no additional repair command is required for legacy DIF metadata.

Every changed registered IGSN is queued automatically for a full-metadata DataCite synchronization after its local transaction completes. The console and CSV report contain the sync-run UUID. Local changes remain intact if an asynchronous DataCite item fails; retry only those failures with:

npm run artisan -- igsn:backfill-legacy-dif-metadata --retry-sync=<sync-run-uuid>

After each applied scope, repeat the same dry run. It should report no further automatic changes; unresolved conflicts, unknown paths, and missing DIF records remain visible for manual follow-up.

Repair legacy IGSN classifications

New legacy imports preserve all supported classifications from every <sample> block in their original order. This includes the Medusa, Sonne273, Earth Shape, and ICDP values covered by issues #1191, #1200, #1202, and #1210. Unknown controlled values remain rejected and are reported without rolling back unrelated DIF metadata.

Existing IGSNs are deliberately skipped by the DataCite importer, but they no longer need to be deleted and reimported to recover classifications. The dedicated command audits every imported IGSN, fetches its DIF metadata from the public legacy portal in batches of at most 100, and only appends missing classifications or fills an empty classification type. Existing values, positions, and non-empty types are never removed or overwritten. The command is a dry run unless --apply is supplied:

npm run artisan -- igsn:backfill-classifications --doi=ICDP5054ES1O201
npm run artisan -- igsn:backfill-classifications --report=storage/app/igsn-classification-dry-run.csv
npm run artisan -- igsn:backfill-classifications --apply --report=storage/app/igsn-classification-applied.csv

Deploy the updated classification catalogs and application code before running the command. Review rejected values, type conflicts, missing DIF documents, and technical errors in the dry-run report before applying changes globally. Use --limit for a bounded run, repeat --doi for selected handles or DOIs, and resume after the last completed Resource ID with --after-id. A successful apply invalidates only changed published landing-page caches. A second global dry run must report no remaining supported classifications as would_update.

The command also repairs still-incomplete classification data from the earlier issues, so their former delete-and-reimport procedure is obsolete. The separate igsn_metadata.user_code schema correction from issue #1192 still requires its existing migration. The rejected vocabulary request from issue #1201 remains intentionally excluded.

Legacy IGSN sample images

When a legacy DIF record contains a sample image, the import stores its validated source description with the IGSN metadata. Known GFZ Data Services images are downloaded only after the metadata transaction has committed and are served from the persistent Laravel public disk. Known ICDP image URLs are normalized to https://data.icdp-online.org/..., checked with a bounded allowlist-protected request, and published as external images only after a valid JPEG response. Definitive missing responses are reported as unavailable and do not produce a public image card; temporary probe failures remain distinct and retryable. Unknown hosts, unsafe paths, placeholders, invalid MIME types, oversized files, and failed downloads likewise do not roll back an otherwise successful metadata import. The completed import dialog reports unavailable images separately from technical failures.

Configure the storage disk and download limits with IGSN_IMAGE_DISK, IGSN_IMAGE_CONNECT_TIMEOUT, IGSN_IMAGE_TIMEOUT, and IGSN_IMAGE_MAX_BYTES. External probes additionally use IGSN_IMAGE_EXTERNAL_PROBE_TIMEOUT and IGSN_IMAGE_EXTERNAL_PROBE_MAX_BYTES, which default to 10 seconds and 256 KiB. The managed-image size limit defaults to 20 MiB and only validated JPEG files are accepted. In production, the selected disk must be persistent, backed up, and publicly linked in the same way as the existing Laravel public disk.

The older image-only backfill deliberately considers only IGSNs that already exist in ERNIE. It remains available for downloading or replacing managed GFZ images, but it is not required to repair the complete Legacy-DIF projection or remove unavailable external ICDP links. It is a dry run unless --apply is supplied:

npm run artisan -- igsn:backfill-images --doi=GFSO273N39
npm run artisan -- igsn:backfill-images --apply --after-id=0 --chunk=100 --report=storage/app/igsn-image-backfill.csv

Use --limit for bounded rollout batches, repeat --doi to select multiple handles or DOIs, and use --force only when already processed images must be revalidated or replaced. A failed run can resume after the last reported resource ID with --after-id. The command is idempotent; missing legacy DIF records and records without an image are reported separately from real processing failures.

IGSN landing-page templates expose every IGSN module, including Sample Image and Location, in a shared two-column editor. Modules can be reordered within a column or moved across columns; each module must occur exactly once across the saved layout. The built-in Templates IGSN copy template places Sample Image in the right column immediately before Location.

Cloned Resource landing-page templates use the same two-column interaction, but expose only Resource modules. Every module can be reordered or moved between columns and must occur exactly once across the complete layout. Description, people, funding, keyword, and metadata-download modules retain the existing shared metadata card in each occupied column; moving them does not create separate cards. The built-in Templates Resources copy template remains immutable and keeps the canonical layout used for new clones.

DataCite landing-page domain migration

The admin-only actions on /resources and /igsns use a persistent queue and the shared application cache. The Docker worker consumes the dedicated datacite queue. Queue connections whose configured driver is sync or null are rejected regardless of the connection name, because the run must survive request timeouts, browser navigation, deployments, and worker restarts.

Keep the safe rate defaults from .env.example: at most 300 authenticated requests per rolling five-minute window, at least one second between requests, concurrency one, a 10-second connection timeout, and a 30-second request timeout. Target landing-page reachability checks have separate configurable connection and total timeouts of three and eight seconds. All DataCite writes in ERNIE share this limiter. Redis or another cache shared by every web and queue process is therefore required; an in-process array cache is not safe outside automated tests.

After a domain move, the configured HTTPS APP_URL is the sole source for every new landing-page URL. There is deliberately no separate old-host or expected-new-host setting. DATACITE_USER_AGENT_EMAIL should identify the operational contact DataCite can reach.

For a local dry run, use DataCite test credentials and DATACITE_TEST_MODE=true, configure APP_URL to the HTTPS base URL being tested, and make sure the generated target URLs are actually reachable from the app/queue containers. Never place production credentials in a local environment.

Troubleshooting

419 Page Expired

  • A plain URL switch to https://localhost:3333 is not enough while ERNIE_DEV_SESSION_DOMAIN=ernie.localhost.
  • For a localhost fallback, set ERNIE_DEV_HOST=localhost and ERNIE_DEV_SESSION_DOMAIN=localhost in .env.docker, keep localhost:3333 in ERNIE_DEV_STATEFUL_DOMAINS, then restart the stack.

public/hot is missing on Windows

Docker Desktop can fail to sync the file back to the host even when it exists in the container.

docker compose --env-file .env.docker -f docker-compose.dev.yml exec vite sh -c 'echo "https://ernie.localhost:3333" > /var/www/html/public/hot'

F-UJI is not reachable

That is expected unless the matching profile was started:

  • npm run docker:dev:assessment
  • npm run docker:dev:parity

For F-UJI specifically, the app still treats the integration as disabled until FUJI_ENABLED=true is set in .env.docker and the stack is restarted.

The first startup is slow

The initial boot may still need to:

  • build images
  • install Composer dependencies
  • install npm dependencies
  • run migrations
  • seed baseline data
  • calculate the initial license usage ranking

Subsequent startups are usually much faster because Docker volumes keep vendor, node_modules, and the MySQL data directory.

The License dropdown is entirely alphabetical

The Data Editor orders active licenses by the persisted rights.usage_count value and uses the license name as the tie-breaker. If every counter is 0, the whole list therefore appears alphabetical.

Check that the scheduler service is running and inspect its startup output. For an immediate diagnostic refresh, run:

npm run artisan -- rights:update-usage-count

Restarting the scheduler also performs this refresh before schedule:work starts. A failed startup refresh leaves the previous complete counter snapshot in place and causes the scheduler container to restart instead of silently serving a partially updated ranking.

MySQL-Sensitive Pest Slice

The default local Pest loop remains SQLite-backed.

Use the dedicated MySQL-backed slice only when driver-sensitive verification is required:

npm run test:php:mysql-sensitive

This command:

  • starts the backend containers if needed
  • verifies and uses the MySQL 9.7 service from docker-compose.dev.yml
  • creates an isolated ernie_test schema inside that local MySQL container
  • runs the current explicit MySQL-sensitive migration file slice with a schema reset before each file

It does not reuse the regular development schema. For broader testing guidance, see testing.md.