Dev-crawler into dev

.gitignore

@@ -0,0 +1 @@
+.idea/

crawler/.dockerignore

@@ -0,0 +1,5 @@
+node_modules
+npm-debug.log
+.git
+.gitignore
+*.md

crawler/.env.example

@@ -0,0 +1,22 @@
+# STAC Crawler Configuration
+# Copy this file to .env and adjust values as needed
+
+# URI of the Postgres/PostGis Host
+PGHOST=example_db_URL
+# Postgres/PostGis Port
+PGPORT=5432
+# Postgres/PostGis user
+PGUSER=example_user
+# Postgres/PostGis password
+PGPASSWORD=example_password
+# Postgres/PostGis databse
+PGDATABASE=example_db
+
+# 'catalogs', 'apis', or 'both'
+CRAWL_MODE=both
+# Maximum Static Catalogs that are crawled (0 ist unlimited) 
+MAX_CATALOGS=0
+# Maximum API Catalogs that are crawled (0 ist unlimited)
+MAX_APIS=0
+# Timeout of connection of crawler
+TIMEOUT_MS=30000

crawler/.gitignore

@@ -0,0 +1,3 @@
+node_modules
+storage
+.env

crawler/Dockerfile

@@ -0,0 +1,17 @@
+# Use official Node.js LTS image
+FROM node:20-alpine
+
+# Set working directory
+WORKDIR /app
+
+# Copy package files
+COPY package*.json ./
+
+# Install dependencies
+RUN npm install --omit=dev
+
+# Copy application files
+COPY . .
+
+# Run the crawler
+CMD ["node", "index.js"]

crawler/README.md

@@ -0,0 +1,156 @@
+# STAC Crawler
+
+A Node.js crawler for STAC Index API that fetches and processes catalog and collection data with configurable options.
+
+## Configuration
+
+The crawler can be configured using environment variables, CLI arguments, or a combination of both. CLI arguments take precedence over environment variables.
+
+### Configuration Options
+
+| Option | CLI Flag | Environment Variable | Default | Description |
+|--------|----------|---------------------|---------|-------------|
+| Mode | `-m, --mode` | `CRAWL_MODE` | `both` | Crawl mode: `catalogs`, `apis`, or `both` |
+| Max Catalogs | `-c, --max-catalogs` | `MAX_CATALOGS` | `10` | Maximum number of catalogs to process |
+| Max APIs | `-a, --max-apis` | `MAX_APIS` | `5` | Maximum number of APIs to process |
+| Timeout | `-t, --timeout` | `TIMEOUT_MS` | `30000` | Timeout per operation in milliseconds |
+| Max Depth | `-d, --max-depth` | `MAX_DEPTH` | `3` | Maximum recursion depth for nested catalogs |
+
+### Using Environment Variables
+
+1. Copy the example environment file:
+```bash
+cp .env.example .env
+```
+
+2. Edit `.env` to customize settings:
+```bash
+CRAWL_MODE=both
+MAX_CATALOGS=20
+MAX_APIS=10
+TIMEOUT_MS=60000
+MAX_DEPTH=5
+```
+
+3. Run the crawler:
+```bash
+npm start
+```
+
+### Using CLI Arguments
+
+Run the crawler with command-line arguments to override defaults or environment variables:
+
+```bash
+# Crawl only catalogs with custom limits
+node index.js --mode catalogs --max-catalogs 20
+
+# Crawl only APIs with extended timeout
+node index.js -m apis -a 10 -t 60000
+
+# Crawl both with all custom settings
+node index.js -m both -c 50 -a 20 -t 45000 -d 5
+```
+
+### Show Help
+
+Display all available options:
+
+```bash
+node index.js --help
+```
+
+## Running Locally
+
+```bash
+# Install dependencies
+npm install
+
+# Run with default configuration
+npm start
+
+# Run with custom configuration
+node index.js --mode catalogs --max-catalogs 15
+```
+
+## Docker
+
+### Build and run with Docker
+
+```bash
+# Build the image
+docker build -t stac-crawler .
+
+# Run with default configuration
+docker run --rm stac-crawler
+
+# Run with environment variables
+docker run --rm -e CRAWL_MODE=apis -e MAX_APIS=10 stac-crawler
+
+# Run with CLI arguments
+docker run --rm stac-crawler --mode catalogs --max-catalogs 20
+```
+
+Or use npm scripts:
+
+```bash
+npm run docker:build
+npm run docker:run
+```
+
+### Using Docker Compose
+
+Create a `.env` file or modify `docker-compose.yml` to set environment variables:
+
+```bash
+# Start the crawler
+docker-compose up -d
+
+# View logs
+docker-compose logs -f
+
+# Stop the crawler
+docker-compose down
+```
+
+Or use npm scripts:
+
+```bash
+npm run docker:compose:up
+npm run docker:compose:down
+```
+
+## Examples
+
+### Example 1: Quick API Test
+Crawl only the first 3 APIs with a short timeout:
+```bash
+node index.js -m apis -a 3 -t 15000
+```
+
+### Example 2: Deep Catalog Exploration
+Crawl 100 catalogs with maximum depth and extended timeout:
+```bash
+node index.js -m catalogs -c 100 -d 10 -t 120000
+```
+
+### Example 3: Balanced Crawl
+Crawl both catalogs and APIs with moderate settings:
+```bash
+node index.js -m both -c 25 -a 15 -t 45000 -d 4
+```
+
+### Example 4: Production Environment
+Set up `.env` for production:
+```bash
+CRAWL_MODE=both
+MAX_CATALOGS=1000
+MAX_APIS=500
+TIMEOUT_MS=60000
+MAX_DEPTH=5
+```
+
+Then run:
+```bash
+npm start
+```