Dev crawler humam into dev - #297

Closed

Mammutor wants to merge 204 commits into

devfrom

dev-crawler-humam

Commits on Nov 12, 2025

Node init
vertrox78
committed
e0b8fbc
Basic Crawling of STAC Index API
Mammutor
committed
ff72c8d
feat: every object in the stac index, will be converted to an dynamic array
Mammutor
committed
30530da

Commits on Nov 18, 2025

added crawling to collection level, including nested catalogs
vertrox78
committed
eb0c8fd
Closes #55; added crawling to collection level, including nested catalogs
vertrox78
committed
69b9ebb
Merge branch 'dev-crawler' of https://github.com/SpatioCore/STAC-Atlas into dev-crawler
vertrox78
committed
65c6cad

Commits on Nov 19, 2025

Commits on Nov 20, 2025

feat: basic api crawling
Mammutor
committed
e15fcae

Commits on Nov 24, 2025

implements #76, stac api validation for catalog crawling
vertrox78
committed
fa5fdae
fix: updated crawler project to be ESM compliant
Mammutor
committed
b611ef4

Commits on Nov 25, 2025

Commits on Nov 26, 2025

Commits on Nov 27, 2025

feat: functions to save crawled catalogs, collections and apis in the database
LennKru
committed
4b818c7

Commits on Nov 28, 2025

Commits on Dec 2, 2025

added `.env`
Sönke Hoffmann
committed
243c351
added environment for docker-compose.yml
Sönke Hoffmann
committed
40422c9
added description of how to use the `.env` and `example.env` in the `README.md`
SonkeHoffmann
committed
b02ad17
changed a few things
SonkeHoffmann
committed
9005ccc
now, everthing should be done.
SonkeHoffmann
committed
eee44db
layout issues fixed
SonkeHoffmann
committed
aecce7e
Fixed Typo/incomplete Sentence in README.md
RobinGummels
authored
7a885ce

Commits on Dec 3, 2025

Commits on Dec 4, 2025

added `stac_id` for collections
SonkeHoffmann
committed
a4ed54f
all IDs are now written in the newer PostgrSQL standart:
SonkeHoffmann
committed
0bd589e
changed `extend` to `extent`.
SonkeHoffmann
committed
49dd720

Commits on Dec 10, 2025

Scheduling and time logic for recrawling (#190)
vertrox78
authored
5f017a9

Commits on Dec 13, 2025

Merge branch 'dev' into dev-database
SonkeHoffmann
authored
f890e93

Commits on Dec 14, 2025

Changed language used in `./api/README.md` from german to english.
RobinGummels
committed
0b396e3

Commits on Dec 17, 2025

Merge branch 'dev' into dev-crawler-Lenn
LennKru
committed
fe7a1a5

Commits on Jan 1, 2026

fix: time utility was missing and improved main module check logic (#193)
Mammutor
authored
eb6d088

Commits on Jan 3, 2026

added triggering function for an auto-update search_vector, both for collections and catalogs.
Sönke Hoffmann
committed
9e9bc30
Merge branch 'dev' into dev-database
SonkeHoffmann
authored
6a8e6c3
changed the CI-Pipeline.
Sönke Hoffmann
committed
b7d51a2

Commits on Jan 7, 2026

bug fix: extent was written wrong
LennKru
committed
9cca397
Merge branch 'dev-crawler-Lenn' of github.com:SpatioCore/STAC-Atlas into dev-crawler-Lenn
LennKru
committed
297a28f
A quick cleanup and fix for the crawler to work for docker (#210)
Mammutor
authored
1e335db
bug fix: extent was written wrong (#212)

authored
24f6cb5

Commits on Jan 8, 2026

Commits on Jan 10, 2026

added different users for the api and crawler groups.
Sönke Hoffmann
committed
c65c348

Commits on Jan 11, 2026

Commits on Jan 13, 2026

feat(crawler): enhance concurrency and storage configuration for crawling
Mammutor
committed
81bd246

Commits on Jan 14, 2026

Commits on Jan 15, 2026

Commits on Jan 16, 2026

Merge branch 'dev-crawler' into dev-crawler-humam
Mammutor
committed
7c28bdc
Enhance memory management in API and handlers
Mammutor
committed
c5b2e8f

Commits on Jan 18, 2026

Refactor database operations in insertOrUpdateCollection and insertSummary functions
Mammutor
committed
9a26ae0

Commits on Jan 19, 2026

Commits on Jan 20, 2026

Commits on Jan 21, 2026

Commits on Jan 22, 2026

Commits on Jan 26, 2026

bug: the type should now be a catalog or collection
LennKru
committed
5e8cb1a

Commits on Jan 27, 2026

Commits on Jan 28, 2026

Commits on Jan 31, 2026

Crawler: is_API=true wenn das gecrawlte eine api ist (#278)
LennKru
authored
1551fbc
changed the whole structure of the catalogs. Now the catalogs can be seen as a crawlog for teh crawler. The crawllog can be now used in case the crawling gets cancelled and the crawler can use the …
Sönke Hoffmann
committed
7a25e89
We asked Mohr on the handling with valid and unvalid collection. We don't need to save the unvalid collections, so also don't need the collumn that points that out
Sönke Hoffmann
committed
724634e
Merge branch 'dev-crawler' into dev-database
SonkeHoffmann
authored
e63b89d
Merge remote-tracking branch 'origin/dev-database' into dev-crawler-humam
Mammutor
committed
2950c53
the reference for the crawllog_collections weren't right
Sönke Hoffmann
committed
0802b00
Dev database: changed catalog handling (#280)

authored
731f9d9
removing duplicate networks
Mammutor
authored
4730c4c
I hate SQL
Sönke Hoffmann
committed
6ae3942
feat(scheduler): enforce time window for crawler execution and add related checks
LennKru
committed
c96d17e
I acciddently deleted keywords and extentions, now they are back
Sönke Hoffmann
committed
de6d974
Merge branch 'dev-crawler' into dev-database
SonkeHoffmann
authored
2429d23
Merge pull request #283 from SpatioCore/dev-database
vertrox78
authored
1a05876
feat(crawler): add scheduler and time window configuration to .env and update scheduler logic
LennKru
committed
a56d4e8
Merge remote-tracking branch 'origin/dev-crawler' into dev-crawler-Lenn
LennKru
committed
57d3840
refactor(scheduler, db): update file overviews and enhance documentation for clarity
LennKru
committed
78c12ee
Merge pull request #287 from SpatioCore/dev-crawler-Lenn
vertrox78
authored
901e517
docs(crawler): enhance README and .env.example with detailed configuration options and usage examples
LennKru
committed
babae45
Merge pull request #289 from SpatioCore/dev-crawler-Lenn
vertrox78
authored
a785f0b
added ER-Diagramm to explain the database structure
Sönke Hoffmann
committed
f878352
updated the README to the newest stand
Sönke Hoffmann
committed
74f96c7
Added Resume and Pause capabilty with the new db layout, edited the test accordingly. Closes #270 (#285)
vertrox78
authored
5c348f1
added explaniation why the migration folder is empty and explained how to use it in the future
Sönke Hoffmann
committed
c83fb72
Merge branch 'dev' into dev-database
SonkeHoffmann
authored
dca10f7
Added LIKE-Operator to CQL2
RobinGummels
committed
372d520
Openapi-Docu plus `context` returned to `GET /collections`
RobinGummels
committed
971dff3
Refactored `README.md`
RobinGummels
committed
83ed69a
Some addtional linting to clean up the script
RobinGummels
committed
6520ab6
Fixed Matched-Count
RobinGummels
committed
2589236
Added parameters `api` and `active` as queryables.
RobinGummels
committed
b55340d
Merge pull request #263 from SpatioCore/dev-api-jonas
BrokeJ
authored
fca8817
Overhaul of API-Examples
RobinGummels
committed
8b3fb72
Merge branch 'dev' into dev-api
RobinGummels
authored
5022e8b
Implemented Load-Testing with `artillery`
RobinGummels
committed
b17ec41
Fixed tests
RobinGummels
committed
69f9f0a

Commits on Feb 1, 2026