Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
204 commits
Select commit Hold shift + click to select a range
e0b8fbc
Node init
vertrox78 Nov 12, 2025
ff72c8d
Basic Crawling of STAC Index API
Mammutor Nov 12, 2025
30530da
feat: every object in the stac index, will be converted to an dynamic…
Mammutor Nov 12, 2025
eb0c8fd
added crawling to collection level, including nested catalogs
vertrox78 Nov 18, 2025
69b9ebb
Closes #55; added crawling to collection level, including nested cata…
vertrox78 Nov 18, 2025
65c6cad
Merge branch 'dev-crawler' of https://github.com/SpatioCore/STAC-Atla…
vertrox78 Nov 18, 2025
eaea3e0
feat: implement database connection with node pg for catalog management
LennKru Nov 19, 2025
62ab362
Merge remote-tracking branch 'origin/dev-crawler' into dev-crawler-Lenn
LennKru Nov 19, 2025
dda02f2
closes #53, set up docker for the crawler component
vertrox78 Nov 19, 2025
b3e289f
fixes #55, restructured the crawler component into catalog and api cr…
vertrox78 Nov 19, 2025
4be5bc8
feat: add dotenv for environment variable management and improved dat…
LennKru Nov 19, 2025
427b299
Merge remote-tracking branch 'origin/dev-crawler' into dev-crawler-Lenn
LennKru Nov 19, 2025
cc32162
deleted .env
LennKru Nov 19, 2025
88f4bcc
updated crawling with use uf stac-js
vertrox78 Nov 19, 2025
e15fcae
feat: basic api crawling
Mammutor Nov 20, 2025
fa5fdae
implements #76, stac api validation for catalog crawling
vertrox78 Nov 24, 2025
b611ef4
fix: updated crawler project to be ESM compliant
Mammutor Nov 24, 2025
9fc2900
refactor: improve catalog handling in crawler by restructuring catalo…
Mammutor Nov 25, 2025
ad9a440
feat: enhance database operations:
LennKru Nov 25, 2025
d0691ba
fix: adjust API crawling limit to include all found APIs instead of t…
Mammutor Nov 25, 2025
3c37635
updated pghost
LennKru Nov 26, 2025
f1f3e7b
feat: changed database connection configuration and add insertStacExt…
LennKru Nov 26, 2025
54a231c
removed the api valication #76
vertrox78 Nov 26, 2025
41a346c
Merge branch 'dev-crawler' of https://github.com/SpatioCore/STAC-Atla…
vertrox78 Nov 26, 2025
c9852fe
feat: implement helper functions for inserting STAC extensions, summ…
LennKru Nov 26, 2025
babb97d
Merge branch 'dev-crawler' into dev-crawler-Lenn
Mammutor Nov 26, 2025
239b5aa
feat: enhance crawler configuration with CLI and environment variable…
vertrox78 Nov 26, 2025
5826775
Merge pull request #156 from SpatioCore/dev-crawler-Lenn
Mammutor Nov 26, 2025
218a409
Merge branch 'dev-crawler' into dev-crawler-jakob
Mammutor Nov 26, 2025
b0825c6
Merge pull request #158 from SpatioCore/dev-crawler-jakob
Mammutor Nov 26, 2025
2c58b9c
Merge remote-tracking branch 'origin/dev' into dev-crawler
Mammutor Nov 26, 2025
4b818c7
feat: functions to save crawled catalogs, collections and apis in the…
LennKru Nov 27, 2025
6c47638
#154, refactored the crawler to use crawlee Framework, improved perfo…
vertrox78 Nov 28, 2025
fdb0cad
#53,example environment configuration file for database and crawler s…
vertrox78 Nov 28, 2025
243c351
added `.env`
Dec 2, 2025
40422c9
added environment for docker-compose.yml
Dec 2, 2025
b02ad17
added description of how to use the `.env` and `example.env` in the `…
SonkeHoffmann Dec 2, 2025
9005ccc
changed a few things
SonkeHoffmann Dec 2, 2025
eee44db
now, everthing should be done.
SonkeHoffmann Dec 2, 2025
aecce7e
layout issues fixed
SonkeHoffmann Dec 2, 2025
7a885ce
Fixed Typo/incomplete Sentence in README.md
RobinGummels Dec 2, 2025
14a6013
converted to es modules, added --no-db mode for debugging and moved c…
vertrox78 Dec 3, 2025
7b7e969
Update .env.example
vertrox78 Dec 3, 2025
72c4f40
Merge pull request #163 from SpatioCore/dev-crawler-jakob
Mammutor Dec 3, 2025
9c01cd8
Merge branch 'dev-crawler' into dev-crawler-Lenn
LennKru Dec 3, 2025
f1f3250
Merge pull request #166 from SpatioCore/dev-crawler-Lenn
Mammutor Dec 3, 2025
86aaacd
Refactor database interactions by moving db helper to utils and imple…
Mammutor Dec 3, 2025
4ac00b0
Merge pull request #168 from SpatioCore/dev-crawler-humam-fixDB
LennKru Dec 3, 2025
166ff8b
#176, fixed logic for max apis, fixed double db init
vertrox78 Dec 3, 2025
a7411ff
#167, fixed logic for max apis, fixed double db init
vertrox78 Dec 3, 2025
c997896
#153, refactor: update API crawling logic to use Crawlee and enhance …
vertrox78 Dec 3, 2025
67a027a
Merge branch 'dev-crawler-jakob' of https://github.com/SpatioCore/STA…
vertrox78 Dec 3, 2025
4103394
Update .gitignore and enhance catalog/collection database handling
LennKru Dec 3, 2025
6429285
psst
vertrox78 Dec 3, 2025
0b47037
Merge pull request #169 from SpatioCore/dev-crawler-jakob
Mammutor Dec 3, 2025
b82b180
Update .gitignore and enhance catalog/collection database handling
LennKru Dec 3, 2025
5f8f03e
Merge branch 'dev-crawler-Lenn' of github.com:SpatioCore/STAC-Atlas i…
Mammutor Dec 3, 2025
29e155d
Merge pull request #171 from SpatioCore/dev-crawler-Lenn
Mammutor Dec 3, 2025
a4ed54f
added `stac_id` for collections
SonkeHoffmann Dec 4, 2025
0bd589e
all IDs are now written in the newer PostgrSQL standart:
SonkeHoffmann Dec 4, 2025
49dd720
changed `extend` to `extent`.
SonkeHoffmann Dec 4, 2025
5f017a9
Scheduling and time logic for recrawling (#190)
vertrox78 Dec 10, 2025
f890e93
Merge branch 'dev' into dev-database
SonkeHoffmann Dec 13, 2025
0b396e3
Changed language used in `./api/README.md` from german to english.
RobinGummels Dec 14, 2025
fe7a1a5
Merge branch 'dev' into dev-crawler-Lenn
LennKru Dec 17, 2025
eb6d088
fix: time utility was missing and improved main module check logic (#…
Mammutor Jan 1, 2026
9e9bc30
added triggering function for an auto-update search_vector, both for …
Jan 3, 2026
6a8e6c3
Merge branch 'dev' into dev-database
SonkeHoffmann Jan 3, 2026
b7d51a2
changed the CI-Pipeline.
Jan 3, 2026
9cca397
bug fix: extent was written wrong
LennKru Jan 7, 2026
297a28f
Merge branch 'dev-crawler-Lenn' of github.com:SpatioCore/STAC-Atlas i…
LennKru Jan 7, 2026
1e335db
A quick cleanup and fix for the crawler to work for docker (#210)
Mammutor Jan 7, 2026
24f6cb5
bug fix: extent was written wrong (#212)
LennKru Jan 7, 2026
cc60a7c
refactor(api): simplify collections link handling in STAC API root fu…
Mammutor Jan 8, 2026
08b398d
chore(crawler): comment out restart policy in docker-compose.yml
Mammutor Jan 8, 2026
e095d0a
fix(crawler): update .env.example and normalize base URL handling
Mammutor Jan 8, 2026
c65c348
added different users for the api and crawler groups.
Jan 10, 2026
e0b2543
chore(db): update docker-compose.yml for network configuration
Mammutor Jan 11, 2026
ef37617
Refactor(database): SQL trigger definitions for catalog and collectio…
Mammutor Jan 11, 2026
d140405
feat(crawler): implement collection flushing to database
Mammutor Jan 11, 2026
77728a7
Merge pull request #217 from SpatioCore/dev-database-humam
SonkeHoffmann Jan 11, 2026
8674091
added source_url for collections and catalogs. Now the full_json does…
Jan 11, 2026
a1269d4
Merge branch 'dev' into dev-database
SonkeHoffmann Jan 11, 2026
20d8fd2
resolved a Problem I had with git by hand cause I didn't found the fu…
Jan 11, 2026
81bd246
feat(crawler): enhance concurrency and storage configuration for craw…
Mammutor Jan 13, 2026
a5a3b78
Merge pull request #219 from SpatioCore/dev-crawler-humam
vertrox78 Jan 14, 2026
3ba13a9
feat(crawler): implement max depth configuration and batch flushing f…
Mammutor Jan 14, 2026
dea62a7
Merge branch 'dev-crawler' into dev-crawler-humam
Mammutor Jan 14, 2026
44b986a
Crawler: implemented rate limiting, and implements some bug fixes (#211)
vertrox78 Jan 14, 2026
fcbac8b
to be stac conform the stac_id is not allowed to throw an error when …
Jan 14, 2026
babd5d9
Merge remote-tracking branch 'origin/dev-crawler' into dev-crawler-Lenn
LennKru Jan 14, 2026
664aaa8
Merge branch 'dev-crawler' into dev-crawler-humam
Mammutor Jan 14, 2026
62d7cc7
Merge pull request #238 from SpatioCore/dev-crawler-humam
vertrox78 Jan 14, 2026
2782266
Merge branch 'dev' into dev-crawler
Mammutor Jan 14, 2026
fe5ae35
feat: add source URL extraction for catalogs and collections
LennKru Jan 15, 2026
acd1180
feat: update insertOrUpdateCatalog to only process catalogs for trave…
LennKru Jan 15, 2026
df81112
merge from dev to dev-crawler-humam and now to dev-crawler (#243)
Mammutor Jan 15, 2026
31a674a
Merge remote-tracking branch 'origin/dev-crawler' into dev-crawler-Lenn
LennKru Jan 15, 2026
27507ed
Refactor: crawler configuration and memory management
Mammutor Jan 15, 2026
e5f6e33
Crawler added source_url to collection_summaries and changed stac_id,…
LennKru Jan 15, 2026
7c28bdc
Merge branch 'dev-crawler' into dev-crawler-humam
Mammutor Jan 16, 2026
c5b2e8f
Enhance memory management in API and handlers
Mammutor Jan 16, 2026
9a26ae0
Refactor database operations in insertOrUpdateCollection and insertSu…
Mammutor Jan 18, 2026
700f840
Merge branch 'dev' into dev-crawler
RobinGummels Jan 19, 2026
8e9d504
Delete .env.example in root
Mammutor Jan 19, 2026
fde14d7
Merge pull request #251 from SpatioCore/Mammutor-patch-2
vertrox78 Jan 19, 2026
77018aa
Merge pull request #250 from SpatioCore/dev-crawler-humam
vertrox78 Jan 19, 2026
70004cb
fix(crawler-db): Enhance collection extent handling in insertOrUpdate…
Mammutor Jan 20, 2026
f53d2e7
fix(crawler-api): update STAC object creation to disable URL migration
Mammutor Jan 20, 2026
23be843
fix(crawler-api): add defensive JSON validation in API and catalog ha…
Mammutor Jan 20, 2026
3ab484c
chore(crawler): update Node.js version in Dockerfile to 20-alpine
Mammutor Jan 20, 2026
c8aec56
fix(crawler): enhance JSON parsing in API and catalog request handlers
Mammutor Jan 20, 2026
392b0e9
fix(crawler): add S3 URL handling and relative URL conversion in API …
Mammutor Jan 20, 2026
5ee15b6
fix(crawler): update Dockerfile to omit development dependencies duri…
Mammutor Jan 20, 2026
ef3c498
feat(crawler): implement parallel crawling for APIs and catalogs
Mammutor Jan 21, 2026
14ff676
fix(crawler): enhance normalization of collection metadata extraction
Mammutor Jan 21, 2026
7357089
fix(crawler): enhance normalization of collection to include addition…
Mammutor Jan 21, 2026
c48840c
fix(crawler): increase max concurrency for improved throughput in API…
Mammutor Jan 21, 2026
657bd0a
fix(crawler): optimize concurrency settings for API and catalog crawlers
Mammutor Jan 21, 2026
edbf614
refactor(crawler): streamline concurrency settings in API and catalog…
Mammutor Jan 21, 2026
9cd68c6
fix(crawler): correct spatial extent format in database insertion
Mammutor Jan 21, 2026
e140246
feat(crawler): add deadlock handling and retry logic for database col…
Mammutor Jan 21, 2026
9fe7298
refactor(crawler): optimize batch sizes and memory management for low…
Mammutor Jan 21, 2026
048183d
Merge pull request #254 from SpatioCore/dev-crawler-humam
vertrox78 Jan 21, 2026
3efd87f
Merge remote-tracking branch 'origin/dev-crawler' into dev-crawler-Lenn
LennKru Jan 21, 2026
890e35e
feat(crawler): integrate global statistics tracking for enhanced metrics
Mammutor Jan 21, 2026
1e6c134
feat(crawler): enhance storage management and logging for API and cat…
Mammutor Jan 21, 2026
6afe0f1
refactor(crawler): optimize memory management and request limits for …
Mammutor Jan 22, 2026
f17ae3e
refactor(crawler): adjust batch sizes and request limits for improved…
Mammutor Jan 22, 2026
5e8cb1a
bug: the type should now be a catalog or collection
LennKru Jan 26, 2026
2cd58b8
refactor(crawler): enhance API and catalog handling with full object …
Mammutor Jan 27, 2026
3b7a6d7
feat(crawler): implement catalog insertion and update functionality i…
Mammutor Jan 27, 2026
15f4be0
feat(crawler): preserve original STAC JSON during normalization and s…
Mammutor Jan 27, 2026
f12e261
refactor(crawler): update database insertion logic to include source_url
Mammutor Jan 27, 2026
72b0168
feat(crawler): enhance source URL handling in collection processing
Mammutor Jan 27, 2026
80684c0
refactor(crawler): update catalog and collection insertion logic to s…
Mammutor Jan 27, 2026
1d3a3df
feat(crawler): improve STAC type determination in normalization process
Mammutor Jan 27, 2026
ce8fc87
Database: Docker config for using the same network, STAC_ID now uniqu…
Mammutor Jan 27, 2026
e44d3ef
Merge branch 'dev' into dev-database
Mammutor Jan 27, 2026
934c1ab
saving catalogs into database, with stac_id, source_url and description
LennKru Jan 27, 2026
21fdfcc
Merge remote-tracking branch 'origin/dev-crawler-humam' into dev-craw…
LennKru Jan 27, 2026
9bb4e9e
feat(crawler): enhance unique stac_id generation and include stac_ver…
LennKru Jan 27, 2026
253bda2
fix(crawler): source_url for catalogs are now completly saved
LennKru Jan 27, 2026
b749125
Merge pull request #261 from SpatioCore/dev-crawler-humam
vertrox78 Jan 28, 2026
d6ad7fc
#241, Add unit tests for normalization and parallel execution utiliti…
vertrox78 Jan 28, 2026
6d5f149
Merge remote-tracking branch 'origin/dev-crawler' into dev-crawler-Lenn
LennKru Jan 28, 2026
fd1d2bc
type is no longer needes, since we only need to save collections
Jan 28, 2026
e38f77d
added is_valid. So we can save every collection be can also be stac_c…
Jan 28, 2026
e67ce0d
revert all latest changes because catalogs dont need to save in the d…
LennKru Jan 28, 2026
9be2c18
#270, added stac node validation for static catalogs from stac indedx…
vertrox78 Jan 28, 2026
9bc8a56
Fix(crawler): insert query to include correct parameters
LennKru Jan 28, 2026
dfc0c06
skip database save for catalogs (#272)
LennKru Jan 28, 2026
de873b1
Database: Added source_url in crawllog (#274)
Mammutor Jan 28, 2026
cdb22c7
Merge branch 'dev-database' into dev-crawler-humam
Mammutor Jan 28, 2026
5dded0a
feat(crawler): implement is_api for collections and add unit tests
LennKru Jan 28, 2026
29439a3
Merge branch 'dev-crawler-Lenn' of github.com:SpatioCore/STAC-Atlas i…
LennKru Jan 28, 2026
310e93e
Merge remote-tracking branch 'origin/dev-crawler' into dev-crawler-Lenn
LennKru Jan 28, 2026
1551fbc
Crawler: is_API=true wenn das gecrawlte eine api ist (#278)
LennKru Jan 31, 2026
7a25e89
changed the whole structure of the catalogs. Now the catalogs can be …
Jan 31, 2026
724634e
We asked Mohr on the handling with valid and unvalid collection. We d…
Jan 31, 2026
e63b89d
Merge branch 'dev-crawler' into dev-database
SonkeHoffmann Jan 31, 2026
2950c53
Merge remote-tracking branch 'origin/dev-database' into dev-crawler-h…
Mammutor Jan 31, 2026
0802b00
the reference for the crawllog_collections weren't right
Jan 31, 2026
731f9d9
Dev database: changed catalog handling (#280)
SonkeHoffmann Jan 31, 2026
4730c4c
removing duplicate networks
Mammutor Jan 31, 2026
6ae3942
I hate SQL
Jan 31, 2026
c96d17e
feat(scheduler): enforce time window for crawler execution and add re…
LennKru Jan 31, 2026
de6d974
I acciddently deleted keywords and extentions, now they are back
Jan 31, 2026
2429d23
Merge branch 'dev-crawler' into dev-database
SonkeHoffmann Jan 31, 2026
1a05876
Merge pull request #283 from SpatioCore/dev-database
vertrox78 Jan 31, 2026
a56d4e8
feat(crawler): add scheduler and time window configuration to .env an…
LennKru Jan 31, 2026
57d3840
Merge remote-tracking branch 'origin/dev-crawler' into dev-crawler-Lenn
LennKru Jan 31, 2026
78c12ee
refactor(scheduler, db): update file overviews and enhance documentat…
LennKru Jan 31, 2026
901e517
Merge pull request #287 from SpatioCore/dev-crawler-Lenn
vertrox78 Jan 31, 2026
babae45
docs(crawler): enhance README and .env.example with detailed configur…
LennKru Jan 31, 2026
a785f0b
Merge pull request #289 from SpatioCore/dev-crawler-Lenn
vertrox78 Jan 31, 2026
f878352
added ER-Diagramm to explain the database structure
Jan 31, 2026
74f96c7
updated the README to the newest stand
Jan 31, 2026
5c348f1
Added Resume and Pause capabilty with the new db layout, edited the t…
vertrox78 Jan 31, 2026
c83fb72
added explaniation why the migration folder is empty and explained ho…
Jan 31, 2026
dca10f7
Merge branch 'dev' into dev-database
SonkeHoffmann Jan 31, 2026
372d520
Added LIKE-Operator to CQL2
RobinGummels Jan 31, 2026
971dff3
Openapi-Docu plus `context` returned to `GET /collections`
RobinGummels Jan 31, 2026
83ed69a
Refactored `README.md`
RobinGummels Jan 31, 2026
6520ab6
Some addtional linting to clean up the script
RobinGummels Jan 31, 2026
2589236
Fixed Matched-Count
RobinGummels Jan 31, 2026
b55340d
Added parameters `api` and `active` as queryables.
RobinGummels Jan 31, 2026
fca8817
Merge pull request #263 from SpatioCore/dev-api-jonas
BrokeJ Jan 31, 2026
8b3fb72
Overhaul of API-Examples
RobinGummels Jan 31, 2026
5022e8b
Merge branch 'dev' into dev-api
RobinGummels Jan 31, 2026
b17ec41
Implemented Load-Testing with `artillery`
RobinGummels Jan 31, 2026
69f9f0a
Fixed tests
RobinGummels Jan 31, 2026
266b672
Enhanced Tests for greater Code-Coverage
RobinGummels Feb 1, 2026
29bd9bb
Crawler: set is_api in collections table (#294)
LennKru Feb 1, 2026
2e9aeac
Merge branch 'dev-crawler' into dev-crawler-humam
Mammutor Feb 1, 2026
dba09f1
Merge remote-tracking branch 'origin/dev-database' into dev-crawler-h…
Mammutor Feb 1, 2026
560af3d
fix(crawler): Enhance collection URL handling in crawler
Mammutor Feb 1, 2026
de9a290
feat(crawler): Implement batch processing for collection URLs not cat…
Mammutor Feb 1, 2026
8424772
feat(crawler): Add functionality to manage collection URLs in DB queue
Mammutor Feb 1, 2026
89a5347
feat(crawler): Enhance URL handling and queue management for collecti…
Mammutor Feb 1, 2026
0d3f89f
feat(crawler): Add configuration options for parallel crawling and re…
Mammutor Feb 1, 2026
5563b6e
feat(crawler): Enhance catalog and API crawling with pending queue ma…
Mammutor Feb 1, 2026
e3f28c0
feat(crawler): Add checks for function existence before database oper…
Mammutor Feb 1, 2026
e9e9ff4
Merge remote-tracking branch 'origin/dev-api' into dev-crawler-humam
Mammutor Feb 1, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
.idea/
1 change: 1 addition & 0 deletions STAC-Atlas
Submodule STAC-Atlas added at dc2577
7 changes: 6 additions & 1 deletion api/.env.example
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ NODE_ENV=development
# Database Configuration (Debian Server)
# Option 1: Use DATABASE_URL (PostgreSQL connection string)
# The api-user is stac_api (read-only)
DATABASE_URL=postgresql://stac_api:[PASSWORD]@atlas.stacindex.org:5432/stac_db
DATABASE_URL=postgresql://stac_api:[PASSWORD]@atlas.stacindex.org:5430/stac_db

# Option 2: Use individual variables (currently active)
DB_HOST=atlas.stacindex.org
Expand Down Expand Up @@ -42,3 +42,8 @@ API_VERSION=1.0.0
MAX_URL_LENGTH=1MB
MAX_HEADER_SIZE=100KB
MAX_BODY_SIZE=10MB

# Rate Limiting Configuration
# Set to "true" to disable rate limiting (useful for load testing)
# WARNING: Never disable rate limiting in production!
DISABLE_RATE_LIMIT=false
Loading
Loading