Test PR - #244
Closed
RobinGummels wants to merge 96 commits into
Closed
Conversation
…nd queryables routes - Added package.json for project dependencies and scripts. - Implemented GET endpoint for collections. - Created conformance endpoint to list supported conformance classes. - Developed landing page for the API with links to collections and documentation. - Added queryables endpoint to return queryable properties for collections. (If i'm correct this can be removed)
we need to use them because Indexes improve query performance by creating data structures that allow faster lookups and filtering. But there is a catch: Indexes speed up reads but slightly slow down writes (INSERT/UPDATE/DELETE), but since reads are more time-critical, we need to use indexes
- Overhaul of first idea landing page - Added some more tests for the required elements in the landingpage-Catalog
…he `GET /` landing page fix(api): enhance STAC API landing page and conformance links - Overhaul of first idea landing page - Added missing conformance links - Added some more tests for the required elements in the landingpage-Catalog
…explanation of the database, an explanation of the structure and a guide on how to start the docker compose file and change the given ports. The comments by @Mammutor and @RobinGummels were solved
…mance endpoint - implemented condormance endpoint
Implemented GET /confomance - Implemented the GET /conformance Endpoint and extracted the conformance-classes into an array that is now stored in /config/conformanceURIS.js and can be used (and is now used by the Landing Page, i edited the index.js accordingly) by both GET / and GET /conformance. Also added a Test that checks whether the Landing Page presents the same conformance URIS as the conformance Endpoint or not.
Co-authored-by: Robin Tammo Gummels <github@gummels.eu>
Co-authored-by: Robin Tammo Gummels <github@gummels.eu>
* Temporary mock data for testing and frontend development * Added API middleware layer for error handling and validation * TODOs ready? pls review * Added API utilities for query parsing, validation, and response formatting * Added swagger and openapi.yaml * Update queryables.js Refactor queryables endpoint into /collections/queryables * Renamed the collections.js file to mocks-collections.js to better reflect its purpose and improve project clarity * changed README "Projektstruktur" * restart from dev-api 22.11..2025 * API: 2.3 Implement Collections List Endpoint done (added explanations as comments in the code) * API: 2.4 Implement Single Collection Endpoint (added explanations as comments in the code) * Update api/routes/collections.js Co-authored-by: Robin Tammo Gummels <github@gummels.eu> * Changed some of the code with the comments on Github (i will finish it tomorrow morning) * Implement most of the feedback and comments (need to talk about some other changes) * Update api/routes/index.js Changed wording from `/collections/queryables` to `/collections-queryables` * Update api/README.md Changed wording from `/collections/queryables` to `/collections-queryables` * Update api/README.md Removed missing folder * Update api/routes/collections.js Removed TODOs from wrong lines * Update api/routes/collections.js Added TODOs * Update api/routes/queryables.js Changed wording from `/collections/queryables` to `/collections-queryables` * Update api/routes/queryables.js Changed wording from `/collections/queryables` to `/collections-queryables` --------- Co-authored-by: VincentKuehn <vkuehn@uni-muenster.de> Co-authored-by: Robin Tammo Gummels <github@gummels.eu>
database ready (with docker)
…#159) * feat(api): add collection search parameters and validation middleware * Added unit-test for validator-functions and integration-tests for `GET /collections`-Querys. - Also minor bugfix, because the validator accepted deecimals as tokens.
* database connection in implementated. The parameters for the connection have to added in the .env-file. Also there is test-file for testing and console messages (installed `pg`) * support for spatial queries via postgis + error handling for datatbase operations changed language to english * error handling * added DATABASE_URL There is an issue with the distance query. Changed the error handling and testing, the console messages are now way better structured * found the Problem with the distance query. The layer are so big, that they reach over the 180° long (PostgGIS can't handel that). Now the calc is done by degree and not meters. * The two files `test-data-retrieval.js` and `verify-schema.js` have been added. `test-data-retrieval` (theoretical, checks against the spezification): ``` Discovers all tables and columns and validates against expected schema. ``` The second files `verify-schema.js` (practical, checks against the real data): ``` Discovers all tables and columns, validates against expected schema ``` * pooling error hanling and log imporoved. renamed tests files to actual test-files * standalone node tests were convertad into JEST * write file `validateRequest.js`. Validates every incoming API request, whether the request is valid and logical. * commented `stac_id` from the tests, it is not in both databases, so the tests for `stac_id` will always fail Added explanation to the `.env.example`, which port is which database * added example pattern for API - database connection. * deleted `validateRequest` cause it's already implemented by @RobinGummels --------- Co-authored-by: Sönke Hoffmann <shoffma5@uni.muenster.de>
* added `.env`
* added environment for docker-compose.yml
now every connection-details are inside an `.env`. There is an `example.env` for better understanding which need to be set as connection details
* added description of how to use the `.env` and `example.env` in the `README.md`
* changed a few things
e.g. DB_PORT --> ${DB_PORT}
* now, everthing should be done.
my god, help. sorry
* layout issues fixed
* Fixed Typo/incomplete Sentence in README.md
---------
Co-authored-by: Sönke Hoffmann <shoffma5@uni.muenster.de>
Co-authored-by: Robin Tammo Gummels <github@gummels.eu>
…match 05_indexes.sql correctly
…test" in the validator.
…to dev-api-vincent
- Renamed `collection.id` to `c.collection.id`
…for collections. Added some tests and fixed some already existing tests, becuase now the tablenames start with the alias `c.`.
…to dev-api-vincent
- needed to do some modifying to the app.js
…e onyl valid to `core`.
…of `primary Keys`
* added `.env`
* added environment for docker-compose.yml
now every connection-details are inside an `.env`. There is an `example.env` for better understanding which need to be set as connection details
* added description of how to use the `.env` and `example.env` in the `README.md`
* changed a few things
e.g. DB_PORT --> ${DB_PORT}
* now, everthing should be done.
my god, help. sorry
* layout issues fixed
* Fixed Typo/incomplete Sentence in README.md
* added `stac_id` for collections
* all IDs are now written in the newer PostgrSQL standart:
```SQL
id SERIAL PRIMARY KEY,
```
changed to
```SQL
id INTEGER PRIMARY KEY GENERATED ALWAYS AS IDENTITY,
```
* changed `extend` to `extent`.
* Changed language used in `./api/README.md` from german to english.
I wanted to thsi anyway at some point, but this is now more like a Test-commit to see if the CI/CD Pipeline triggers...
---------
Co-authored-by: Sönke Hoffmann <shoffma5@uni.muenster.de>
Co-authored-by: Robin Tammo Gummels <github@gummels.eu>
API is now responding with all necessary fields for each collection
…ection" (#195) Reverts #185 @SonkeHoffmann accidentally didn't squash correctly.
…ach collection"" (#185) (#195) (#196) dev-api: prepare v1.1.0 + API docs + query builder fixes - Change API version to 1.1.0 - Add OpenAPI spec so /api-docs works locally - Document stac-api-validator usage - Update api/.env.example - Query builder: select required fields for collections across db_tables; adjust tests (alias `c.`) Commits included: - 34bf962 Changed API-Version name to 1.1.0 instead of 1.0.0 - b047389 Added description on how to use `stac-api-validator` (currently only valid for `core`) - b811288 Added `openapi.yaml` (so http://localhost:3000/api-docs/ works); modified app.js accordingly - 6e5ab3e Merge branch 'dev-api-robin' of github.com:SpatioCore/STAC-Atlas into dev-api-robin - 70dc043 Updated Query-Builder to get all necessary fields from all db_tables for collections. Added tests and fixed existing tests (table names now start with alias `c.`) - d83eeb4 Update api/.env.example - 5a7af5b Updated Query-Builder to get all necessary fields from all db_tables for collections. Added tests and fixed existing tests (table names now start with alias `c.`) Co-authored-by: Robin Tammo Gummels <github@gummels.eu>
i encoded the "-1" value in the negative ID test instead of directly putting it into the path.
…ext search (#200) * Add provider and license filters to collection search API - Updated buildCollectionSearchQuery to include provider and license parameters for filtering collections. - Enhanced validateCollectionSearchParams middleware to validate provider and license query parameters. - Modified collections route to handle new provider and license filters in search queries. - Implemented validation functions for provider and license parameters in collectionSearchParams. * Add validation tests for provider and license * Enhance full-text search by including keywords in the tsvector expression and update related tests * Add provider and license to query parameter extraction in collection search validation * Revert "Enhance full-text search by including keywords in the tsvector expression and update related tests" This reverts commit 872443d.
…lder integration (#186) * added SQLQuery-builder with these parameters: q,bbox,datetime,sortby,limit and token * finalised bbox and datetime * adapted to DB, QueryBuilder and added helperfunction runQuery * added question-TODOs * added bbox+datetime to the Query-Builder from Jonas * added tests for Query-Builder from Jonas * added tests from George * added falsely deleted TODOs again * fixed collumn names to match our DB and adjusted full text search to match 05_indexes.sql correctly * Used a formatter and linter on `buildCollectionSearchQuery.js * Did some major and minor fixes to the collection search. - Updated `buildCollectionSearchQuery` to support pagination and improved text search with English language settings. - Modified tests in `buildCollectionsSearchQuery.basic.test.js`, `collections-pagination.test.js`, and `collections-sort.test.js` to reflect new query behavior and validation logic. - Enhanced sort validation in `validators.test.js` and `collectionSearchParams.js` to map API fields to database column names. - Implemented total count retrieval for matched results in `collections.js`. * Added a internal .env creation in the CI/CD Pipeline. It utilzes GitHub Repository Secrets to not publish any private Logins and stuff. * Forgot that the second Job of the CI/CD pipeline runs seperatly and needs a internal .env file too. * Enhance documentation for buildCollectionSearchQuery Updated the documentation for: - the buildCollectionSearchQuery function - the fulltextsearch * Refactor buildCollectionSearchQuery and updated SELECT part Changed the SELECT part to match our bid and the database shema. Updated comments and for clarity. Changed full-text search to use 'simple' configuration instead of 'english'. * Update api/routes/collections.js small typo Co-authored-by: Robin Tammo Gummels <github@gummels.eu> * Remove sorting TODO from collections route Removed TODO comment about sorting based on sortby parameter. * Explicitly return undefined for normalized in validateSortby Update validateSortby function to explicitly return undefined for normalized when sortby is not provided. * small fix in buildCollectionSearch.fulltext.test.js Change plainto_tsquery language from 'english' to 'simple' * Fix duplicate SELECT keyword in query Remove duplicate 'SELECT' keyword in SQL query. * Fix missing newline at end of collectionSearchParams.js * Fixed missing bracket in collectionSearchParams.js * Refactor validateSortby for optional parameter handling Refactor validateSortby function to handle optional sortby parameter and improve validation logic. * Stabilize API test pipeline by running Jest in-band with extended timeout Run Jest in CI with --runInBand and a higher default --testTimeout to stabilize database-backed integration tests. Multiple Jest workers were competing for the same PostgreSQL connection pool and some long-running /collections queries exceeded the default 5s timeout, causing failures in existing test suites (e.g. collectionSearch and DBconnection). * Fixed leaking tests that blocked CI/CD-Pipeline. - Added a global Teardown for jest and force-exited the tests to prevent leaking. - Made a change to db_APIconnection to only log the pool-(dis)connection if it isn't run in a test enviroment. * Did a minimum amount of Formatting to the discription * Used `npm audit fix --force` to fix all vulnerabilties in our used packages. * Fixed curious doublechecking for empty Strings for the sortby-Parameter. - Now we only check once for a empty sortby - And added a test which distinguish between `sortby=""` and `sortby="+"` * Update api/routes/collections.js Removed the TODO about switching from mock-data to the real db * Removed globalTeardown as i brought up some problems corresponding to long db-queries (for example BBOX). Instead i increased the maximal testTimeout. * added validator for collections{id} and correctly implemented collections{id} * added test for collections{id} * removed unnecessary parameter * added id parameter to the Query (temporary fix) * test-fixes to match our current tests and a fix to the baseURL for collection{id} * test fix * fixed problem with tests in api.test.js and adjusted the "invalid-id-test" in the validator. * Update api/routes/collections.js - Renamed `collection.id` to `c.collection.id` * added test for negative ids * deleted the whole "existing links" part and build base Links * fixed bug in validateCollectionId.js * Refactor negative ID test i encoded the "-1" value in the negative ID test instead of directly putting it into the path. * Removed a german comment in `api/routes/collections.js` --------- Co-authored-by: Robin Tammo Gummels <github@gummels.eu>
) This commit implements comprehensive CQL2 (Common Query Language 2) filtering support for the STAC Atlas Collection Search API, enabling advanced queries on collection metadata. ## New Features ### CQL2 Parser Integration - Integrated cql2-wasm (Rust compiled to WebAssembly) for parsing CQL2 - Support for both CQL2-Text and CQL2-JSON encodings - Dynamic ESM import to maintain Jest compatibility with CommonJS ### Basic CQL2 Operators - Comparison operators: =, <, >, <=, >=, <> - Logical operators: AND, OR, NOT - Advanced comparison: BETWEEN, IN, IS NULL ### Spatial Operators (PostGIS) - S_INTERSECTS: Find collections whose geometry intersects with GeoJSON - S_WITHIN: Find collections completely within a geometry - S_CONTAINS: Find collections containing a geometry - Uses ST_GeomFromGeoJSON for geometry parsing ### Temporal Operators - T_INTERSECTS: Find collections with overlapping temporal extents - T_BEFORE: Find collections before a timestamp - T_AFTER: Find collections after a timestamp - Support for open-ended intervals (..) ### Column Mappings - Maps CQL2 properties to database columns with table aliases - Core fields: id, title, description, license, type, etc. - Aggregated fields: keywords, stac_extensions, providers, assets, summaries - Fallback to JSONB full_json column for custom properties ## Files Added or Modified - utils/cql2.js: WASM initialization and CQL2 parsing wrapper - utils/cql2ToSql.js: CQL2 JSON AST to PostgreSQL WHERE clause converter - middleware/validateCollectionSearch.js: Request validation with filter support - docs/cql2-filtering.md: Comprehensive CQL2 documentation - routes/collections.js: Integrated CQL2 filter processing - utils/buildCollectionSearchQuery.js: Added cqlWhere parameter support - config/conformanceURIS.js: Added all CQL2 conformance class URIs - README.md: Added CQL2 section and updated implementation status ## Tests Added - __tests__/cql2ToSql.test.js: Unit tests for SQL conversion (17 tests) - __tests__/cql2.integration.test.js: Integration tests with database (18 tests) - __tests__/buildCollectionSearchQuery_cql.test.js: Query builder CQL2 tests ## Technical Notes ### ESM Compatibility The cql2-wasm package is an ES Module. To maintain compatibility with Jest (CommonJS), the module is loaded via dynamic import() instead of require(). This allows the WASM to be initialized lazily when first needed. ### SQL Injection Prevention All CQL2 filters are converted to parameterized queries with $1, $2, etc. placeholders. Values are passed separately to pg-pool, preventing injection. ## Conformance Classes Implemented - http://www.opengis.net/spec/cql2/1.0/conf/basic-cql2 - http://www.opengis.net/spec/cql2/1.0/conf/advanced-comparison-operators - http://www.opengis.net/spec/cql2/1.0/conf/cql2-json - http://www.opengis.net/spec/cql2/1.0/conf/cql2-text - http://www.opengis.net/spec/cql2/1.0/conf/basic-spatial-functions - http://www.opengis.net/spec/cql2/1.0/conf/spatial-functions - http://www.opengis.net/spec/cql2/1.0/conf/temporal-functions ## Dependencies Added - cql2-wasm@0.4.2: WASM-based CQL2 parser from cql2-rs
- Introduced a new CI job for STAC API validation in the GitHub Actions workflow. - Updated collection retrieval endpoints to support both numeric and string IDs. - Improved validation middleware for collection IDs to ensure proper formatting and length. - Enhanced test cases for collection endpoints to reflect new validation rules and response structures. - Added documentation for STAC API Validator results.
…ing incl. request tracking - all according to RFC 7807 (#213) * feat(api): implement RFC 7807 error handling with request tracking Implement standardized error responses and global error handler to improve API error reporting and debugging capabilities. Resolves API 7.1 (Implement Error Response Format) #119 Resolves API 7.3 (Implement Global Error Handler) #121 Changes: - Add RFC 7807 Problem Details error response format * Standard fields: type, title, status, detail, instance, requestId * Backwards compatibility: maintained code/description fields * Error type URIs: https://stacspec.org/errors/{code} - Implement request ID tracking system * UUID v4 generation for request tracing * Support for client-provided X-Request-ID header * Request ID included in all error responses - Add global error handler with intelligent logging * Severity-based logging (500+: full details, 400+: basic info) * Error message sanitization (removes passwords, tokens, secrets) * Production-safe error messages - Update error responses across codebase * validateCollectionSearch: InvalidParameterValue errors * validateCollectionId: InvalidParameter errors * collections route: NotFound errors * 404 handler: throw errors instead of direct response - Add comprehensive error handler test suite * RFC 7807 compliance validation * Request ID generation and propagation * Error code consistency checks * Message sanitization verification * Removed old mock-data `/api/data/collections.js` as it is no longer used
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
6 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.