Reconcile stable master into dev

.github/workflows/docker.yml

@@ -23,9 +23,18 @@ jobs:
           extra_nix_config: |
             experimental-features = nix-command flakes
 
-      - name: Run image, health, FTS, and persistence tests
+      - name: Run image, health, authentication, FTS, and persistence tests
         run: ./scripts/stack-test.sh
 
+      - name: Upload sanitized stack diagnostics
+        if: always()
+        uses: actions/upload-artifact@v4
+        with:
+          name: stack-test-diagnostics
+          path: stack-test-artifacts/
+          if-no-files-found: error
+          retention-days: 7
+
       - name: Publish images
         if: github.event_name == 'push' && github.ref == 'refs/heads/master'
         env:

.github/workflows/starintel-schema-lock.yml

@@ -0,0 +1,29 @@
+name: Canonical StarIntel schema lock
+
+on:
+  pull_request:
+  push:
+    branches:
+      - master
+      - dev
+      - "agent/**"
+    paths:
+      - "schema/starintel-schema.lock.json"
+      - "scripts/check-starintel-schema-lock.py"
+      - ".github/workflows/starintel-schema-lock.yml"
+  workflow_dispatch:
+
+permissions:
+  contents: read
+
+jobs:
+  research-node-schema:
+    runs-on: ubuntu-latest
+    timeout-minutes: 10
+    steps:
+      - uses: actions/checkout@v6
+      - uses: actions/setup-python@v6
+        with:
+          python-version: "3.13"
+      - name: Verify canonical research-node schema
+        run: python scripts/check-starintel-schema-lock.py

README.org

@@ -1,119 +1,207 @@
-#+title: Readme
+#+title: StarIntel Server
+#+options: toc:2
 
-* Star Server
-Starintel server is the new API server for interacting with the starintel system.
+* StarIntel Server
 
+StarIntel Server is the Common Lisp runtime for storing, routing, querying, and
+recursively enriching StarIntel documents.
 
+It combines:
 
-- Features:
-  - rabbitmq
-  - actor model
-  - hackable
+- CouchDB for durable documents, views, and full-text search.
+- RabbitMQ topic exchanges for document and actor traffic.
+- Sento actors for local concurrency, supervision boundaries, timers, and
+  message passing.
+- A Ningle/Clack/Hunchentoot HTTP API.
+- The =star-cl= document library and StarIntel specification adapters.
+- Nix-built binaries, tests, and container images.
 
+This repository is an experimental operator system, not a hardened public SaaS
+service.
 
-** *WARNINGS*
-*** Project status
-This repo is a mess!
-This project is experimental, DO NOT EXPOSE TO THE WEB.
+#+begin_quote
+*Do not expose the HTTP API or RabbitMQ directly to the public Internet.*
 
-*** Consumers and actors
+The HTTP API currently has no authentication or authorization and sends
+=Access-Control-Allow-Origin: *=. Put it behind an authenticated reverse proxy,
+restrict network access, and treat the Lisp init file as trusted executable
+code.
+#+end_quote
 
-In a future version the consumer that inserts will revert back to cl-gserver actor when I figure a good proto and learn more about sento.
+** What the server does
 
-Something along the lines of
+A document normally moves through this pipeline:
 
-#+begin_src lisp
-(tell *couchdb* (:db "starintel" :id "0HJY....." :document (as-json (spec:create-user :dataset "github" :name "lost-rob0t"))))
-#+end_src
-
-I will have to do more research to better utlize the sento system.
+#+begin_example
+HTTP/client/actor
+      |
+      v
+documents.ingest.<dtype>
+      |
+      v
+CouchDB insert + _id/_rev enrichment
+      |
+      v
+documents.new.<dtype>
+      |
+      +--> local actor via TELL
+      +--> remote actor via RabbitMQ topic route
+      +--> derived documents and relations
+                 |
+                 +--> documents.ingest.<dtype>  (durable recursion)
+                 `--> documents.new.<dtype>     (event-only fan-out)
+#+end_example
 
-** Documentation
-*** Containers
-See [[file:DOCKER.md][the Nix-built Compose stack guide]] for image builds,
-secret setup, migration, backup, and upgrades.
-*** HTTP
-For http documentation: [[file:./docs/http-api-docs.org][Api Documentation]]
-*** Setting up dev env
-star-server uses nixpkgs to managment the dev shell, which is like a venv for this project.
+Actors can emit more StarIntel documents, relations, targets, and actor events.
+Those outputs can trigger more actors. This is *dataflow recursion*: the graph
+expands through messages rather than recursive function calls.
 
-it is not required
+** Quick start: Nix-built Compose stack
 
-First install the [[https://nixos.org/download/][Nix package manager]] and [[https://direnv.net/][Direnv]].
+Requirements: Nix with flakes, Docker Engine, Docker Compose v2, =curl=, =jq=,
+and =openssl=.
 
-Also Ensure you have [[https://www.quicklisp.org/beta/][Quicklisp]] installed also.
+#+begin_src sh
+cp .env.example .env
+install -d -m 0700 secrets
+openssl rand -base64 32 > secrets/couchdb_password
+openssl rand -base64 48 > secrets/couchdb_secret
+openssl rand -hex 24 | tr '[:lower:]' '[:upper:]' > secrets/erlang_cookie
+openssl rand -base64 32 > secrets/rabbitmq_password
+chmod 0600 secrets/*
 
-#+Name: Setup dev env
-#+begin_src sh :async :results output replace
-git clone  --recurse-submodules https://github.com/lost-rob0t/starintel-gserver.git star-server && cd star-server;
-direnv allow .
+nix run .#load-images
+docker compose up --detach --wait
+curl --fail http://127.0.0.1:5000/health
 #+end_src
 
-Now nix should be pulling down everything and you will be placed inside a new shell with everything needed in $PATH.
-*** Compiling From Source without nix
+Default local endpoints:
+
+| Service             | Address                  |
+|---------------------+--------------------------|
+| StarIntel HTTP API  | http://127.0.0.1:5000    |
+| CouchDB             | http://127.0.0.1:5984    |
+| RabbitMQ AMQP       | 127.0.0.1:5672           |
+| RabbitMQ management | http://127.0.0.1:15672   |
+
+See [[file:DOCKER.md][DOCKER.md]] for image builds, secrets, persistence, migration, backup, FTS,
+and upgrades.
+
+** Submit a document
+
+The body must contain a =dtype=. The route chooses the RabbitMQ routing key but
+does not currently inject or validate the body dtype.
+
+#+begin_src sh
+curl --fail \
+  --header 'Content-Type: application/json' \
+  --request POST \
+  --data '{
+    "_id": "example-note",
+    "dataset": "demo",
+    "dtype": "note",
+    "sources": ["manual"],
+    "version": "0.8.0",
+    "dateAdded": 0,
+    "dateUpdated": 0,
+    "content": "first document"
+  }' \
+  http://127.0.0.1:5000/new/document/note
+#+end_src
 
-Ensure you have [[https://www.quicklisp.org/beta/][Quicklisp]].
+The API acknowledges queue publication, not CouchDB persistence. Read the
+document after the ingest consumer has processed it:
 
-#+Name: Compile from source
-#+begin_src shell :async :results output replace
-git clone  --recurse-submodules https://github.com/lost-rob0t/starintel-gserver.git star-server && cd star-server;
-make build
-make install
+#+begin_src sh
+curl --fail http://127.0.0.1:5000/document/example-note | jq
 #+end_src
-*** Usage
 
-#+Name: Usage
-#+begin_src sh :async :results output replace
-./star-server
+** Build and test
+
+#+begin_src sh
+nix build
+nix run .#star-unit-tests
 #+end_src
 
-#+RESULTS: Usage
-#+begin_example
-NAME:
-  star-server - Starintel unified API and document consuming service.
+With CouchDB and RabbitMQ available:
 
-USAGE:
-  star-server [global-options] [<command>] [command-options] [arguments ...]
+#+begin_src sh
+nix run .#star-integration-tests
+#+end_src
 
-OPTIONS:
-      --help     display usage information and exit
-      --version  display version and exit
+Full Nix image, health, FTS, restart, and persistence test:
 
-COMMANDS:
-  start  start the server
+#+begin_src sh
+./scripts/stack-test.sh
+#+end_src
 
-AUTHORS:
-  nsaspy <nsaspy@airmail.cc>
+See [[file:docs/testing.md][docs/testing.md]].
 
-LICENSE:
-  GPL v3
+** Run from Common Lisp
 
-#+end_example
+The pinned Nix build is the supported reproducible path. For interactive
+development:
 
+#+begin_src sh
+nix develop
+sbcl --load run.lisp
+#+end_src
 
-#+Name: start
-#+begin_src shell :async :results output replace
-./star-server start --help
+Build the executable:
+
+#+begin_src sh
+nix build
+./result/bin/star-server start --init ./example_configs/init.lisp
 #+end_src
 
-#+RESULTS: start
+The executable accepts:
+
 #+begin_example
-NAME:
-  star-server start - start the server
+star-server start -i PATH
+star-server start --init PATH
+#+end_example
 
-USAGE:
-  star-server [global-options] start [options] [arguments ...]
+The same path can be supplied through =STAR_SERVER_INIT_FILE=.
 
-OPTIONS:
-      --help              display usage information and exit
-      --version           display version and exit
-  -d, --debugger <VALUE>  Enable Remote debugging
-  -i, --init <VALUE>      Path to init file [default: ./init.lisp] [env: $STAR_SERVER_INIT_FILE]
+** Documentation map
 
-AUTHORS:
-  nsaspy <nsaspy@airmail.cc>
+| Document | Contents |
+|----------+----------|
+| [[file:docs/index.org][docs/index.org]] | Documentation index and implementation status |
+| [[file:docs/architecture.org][docs/architecture.org]] | Runtime structure, startup order, concurrency, and repository layout |
+| [[file:docs/actors.org][docs/actors.org]] | Creating, registering, targeting, scheduling, and operating actors |
+| [[file:docs/messaging.org][docs/messaging.org]] | RabbitMQ exchanges, queues, routing keys, recursion, delivery, and loop control |
+| [[file:docs/document-spec.org][docs/document-spec.org]] | StarIntel 0.9 and legacy 0.8 documents, types, relations, IDs, metadata, and provenance |
+| [[file:docs/configuration.org][docs/configuration.org]] | Environment, init files, secrets, advanced examples, and tuning |
+| [[file:docs/http-api-docs.org][docs/http-api-docs.org]] | HTTP endpoint reference and examples |
+| [[file:DOCKER.md][DOCKER.md]] | Nix-built container stack and operations |
+| [[file:docs/testing.md][docs/testing.md]] | Unit, integration, and stack tests |
 
-LICENSE:
-  GPL v3
+** Runtime status
 
-#+end_example
+The documentation distinguishes three states:
+
+- *Active*: loaded by =source/starintel-gserver.asd= and started by =star::main=.
+- *Present but not active*: code exists in the repository but is not loaded by
+  the ASDF system or is not started by the current startup path.
+- *Stub/experimental*: API or actor code exists but is incomplete.
+
+Important current limits:
+
+- HTTP ingestion does not enforce the strict StarIntel 0.9 schema.
+- The server still uses legacy flat 0.8 constructors in parts of the actor code.
+- =source/actor-systems/user-finder.lisp= and =user-hunt.lisp= are not loaded by
+  the ASDF system.
+- The matcher actor framework is experimental; the URL extractor is loaded, but
+  a complete global pattern-dispatch loop is not wired.
+- =/new/event/:id= is a stub.
+- =*http-api-base-path*= and the HTTP certificate/key variables are not applied
+  by =start-http-api=.
+- The second =/dataset-size= route definition replaces or shadows the first,
+  depending on Ningle route behavior.
+
+These are documented facts, not supported guarantees.
+
+** License
+
+See [[file:LICENSE][LICENSE]].

docker-compose.yml

@@ -99,9 +99,16 @@ services:
       RABBITMQ_USER: ${RABBITMQ_USER:-starintel}
       RABBITMQ_PASSWORD_FILE: /run/secrets/rabbitmq_password
       HTTP_API_LISTEN_ADDRESS: 0.0.0.0
+      STAR_AUTH_MODE: api-key
+      STAR_AUTH_DATABASE: ${STAR_AUTH_DATABASE:-starintel-gserver-auth}
+      STAR_AUTH_PEPPER_FILE: /run/secrets/auth_pepper
+      STAR_AUTH_BOOTSTRAP_SECRET_FILE: /run/secrets/auth_bootstrap_secret
+      STAR_AUTH_ALLOWED_ORIGINS: ${STAR_AUTH_ALLOWED_ORIGINS:-}
     secrets:
       - couchdb_password
       - rabbitmq_password
+      - auth_pepper
+      - auth_bootstrap_secret
     ports:
       - "${STAR_SERVER_BIND_ADDRESS:-127.0.0.1}:${STAR_SERVER_PORT:-5000}:5000"
     networks:
@@ -127,6 +134,10 @@ secrets:
     file: ${CREDENTIALS_DIR:-./secrets}/erlang_cookie
   rabbitmq_password:
     file: ${CREDENTIALS_DIR:-./secrets}/rabbitmq_password
+  auth_pepper:
+    file: ${CREDENTIALS_DIR:-./secrets}/auth_pepper
+  auth_bootstrap_secret:
+    file: ${CREDENTIALS_DIR:-./secrets}/auth_bootstrap_secret
 
 volumes:
   clouseau_index:

docker/star-server-entrypoint.sh

@@ -7,13 +7,23 @@ load_secret() {
   eval "file=\${$file_variable:-}"
 
   if [ -n "$file" ]; then
+    if [ ! -r "$file" ]; then
+      printf '%s\n' "Secret file for ${name} is not readable: ${file}" >&2
+      exit 1
+    fi
     value="$(cat "$file")"
+    if [ -z "$value" ]; then
+      printf '%s\n' "Secret file for ${name} is empty: ${file}" >&2
+      exit 1
+    fi
     export "$name=$value"
   fi
 }
 
 load_secret COUCHDB_PASSWORD
 load_secret RABBITMQ_PASSWORD
+load_secret STAR_AUTH_PEPPER
+load_secret STAR_AUTH_BOOTSTRAP_SECRET
 
 exec su-exec 65532:65532 \
   /bin/star-server start -i /etc/starintel/init.lisp