Add comprehensive docs, secret template, and README cleanup; un-ignore secret.example.txt

.gitignore

@@ -7,6 +7,7 @@ __pycache__
 *.webm
 *.txt
 *secret*
+!secret.example.txt
 *.json*
 *.jpg
 /downloaded_video.mp4.in

README.md

@@ -1,83 +1,63 @@
 # turbo-bot
 
-## How this works
+TurboBot is a Python Signal bot that runs against the signal-cli REST API. It routes messages through dynamically discovered handlers in `handlers/`, supports hashtag commands such as `#gpt`, and can return text plus base64-encoded attachments.
 
-A docker compose file is used to run a signalbot, and the signal rest api.
+## Documentation
 
-The signalbot docker will automatically get this repo and execute run.py.
+- [Architecture](docs/ARCHITECTURE.md): runtime flow, dispatch, handler discovery, and important paths.
+- [Configuration](docs/CONFIGURATION.md): required environment variables and `secret.txt` setup.
+- [Operations](docs/OPERATIONS.md): Docker Compose services, bootstrap flow, auto-updates, and Signal linking.
+- [Writing handlers](docs/HANDLERS.md): how to add new bot features.
+- [GPT function tools](docs/GPT_TOOLS.md): how to add tools callable by the `#gpt` handler.
+- [Testing](docs/TESTING.md): local test commands, mocks, and integration-test caveats.
+- [README migration checklist](docs/README_MIGRATION.md): where the original README content moved and why no setup guidance was dropped.
 
-A second signalbot docker is made that checks out the devel branch instead of main.
+## Quick start
 
-Both containers monitor the github repo and will automatic download updated code.
+1. Start the Docker Compose stack.
+2. Link `signal-cli` to a Signal account/device if this is the first run.
+3. Copy the sample secret file and edit it with real values:
 
+   ```bash
+   cp secret.example.txt secret.txt
+   ```
 
-## Running
+4. Configure at least:
 
-Execute the docker compose file.  In the signalbot docker make sure you make a secrets.txt file that has these vairables:
-
-```
-export      SIGNAL_API_URL=signal-cli:8181 # URL for the signal-cli API
-export      BOT_NUMBER="+1555555555" # The registered Signal number for your bot
-export      CONTACT_NUMERS="+1555555555" # true/false, a single contact, a ; seperated list of contacts
-export      GROUP_NAMES="MYGROUP" # true/false, a single group, a ; seperated list of groups
-export      IGNORE_GROUPS="TurboBot Devel"	#optional
-export      INSTA_USERNAME="myuser"
-export      INSTA_PASSWORD="mypassword"
-export      OPENAI_API_KEY="keygoeshere"
-```
-
-Update the docker-compose file to point the signal-cli bot(s) to your repo,
-or use this one.  Default file makes one for main branch and one for devel
-branch.  The run.sh script will just fail back to bash if you dont supply
-a secrets.sh file in the same folder as the repo.
-
-When you first run this you need to boot signal-cli in normal mode in order
-to link it to your account.  You do that by having it generate a qr code
-that you scan with your phone.  See the signalbot documentation.  tldr:
-http://localhost:8181/v1/qrcodelink?device_name=local
+   ```bash
+   export SIGNAL_API_URL="signal-cli:8181"
+   export BOT_NUMBER="+15555555555"
+   export CONTACT_NUMBERS="+15555555555"
+   export GROUP_NAMES="My Signal Group"
+   ```
 
+See [Configuration](docs/CONFIGURATION.md) for all supported variables.
 
-Manually running signal-cli from command line.  Be sure to stop the instance
-first:
+## Signal linking
 
-```
-docker run --env "MODE=json-rpc" --env "PORT=8181" --env "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin" --env "GIN_MODE=release" --env "BUILD_VERSION=0.90" --env "SIGNAL_CLI_CONFIG_DIR=/home/.local/share/signal-cli" --env "SIGNAL_CLI_UID=1000" --env "SIGNAL_CLI_GID=1000" --entrypoint "/entrypoint.sh" --volume "/share/CACHEDEV1_DATA/Container/container-station-data/lib/docker/volumes/app-1_signal-cli-data/_data:/home/.local/share/signal-cli"  bbernhard/signal-cli-rest-api:latest
-```
+When you first run this, start signal-cli and link it to your account by scanning a QR code with your phone. With the REST API exposed locally, the QR-code endpoint is typically:
 
-This was generated with:
-
-```
-container_id="signal-cli"
-docker inspect $container_id | jq -r '
-  .[] | 
-  "docker run " +
-  (if .Config.Env then (.Config.Env | map("--env \"" + . + "\"") | join(" ")) else "" end) + " " +
-  (if .Config.Entrypoint then "--entrypoint \"" + (.Config.Entrypoint | join(" ")) + "\" " else "" end) +
-  (if .Mounts then (.Mounts | map("--volume \"" + .Source + ":" + .Destination + "\"") | join(" ")) else "" end) + " " +
-  (if .Config.Cmd then (.Config.Cmd | join(" ")) else "" end) +
-  " " + .Config.Image
-'
+```text
+http://localhost:8181/v1/qrcodelink?device_name=local
 ```
 
+See [Operations](docs/OPERATIONS.md) for more deployment details.
 
-# Development
-
-You can develop in windows, linux, wsl, and mac all relatively easily.
-
-For Windows I'd suggest using miniconda.
-
-Running the tests on your own windows or linux box is pretty easy.
+## Development
 
-If on windows just install miniconda, make a new environment, `conda install python==3` and then run `pip3 install -r requirements.txt`.
+Create a Python environment, install dependencies, and run tests:
 
-TODO: we should really match the version of python on the real system... which i just realized we dont control
-
-Then you can run all the tests with `python -m unittest discover -s tests -p "test_*.py"`
-
-You can also do it in WSL 1 or 2.
+```bash
+pip install -r requirements.txt
+python -m unittest discover -s tests -p "test_*.py"
+```
 
-Some handlers might also require apt-get packages.  i.e. ffmpeg.  You can get 
-ffmpeg in miniconda (i.e. conda install ffmpeg) or wsl (via apt-get or whatever).
+Some handlers require system packages, especially `ffmpeg`. Docker deployments install packages from `pkglist`.
 
+## Manual signal-cli reference
 
+If you need to reproduce a running `signal-cli` container manually, stop the existing instance first and adapt this example to your local volume paths:
 
+```bash
+docker run --env "MODE=json-rpc" --env "PORT=8181" --env "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin" --env "GIN_MODE=release" --env "BUILD_VERSION=0.90" --env "SIGNAL_CLI_CONFIG_DIR=/home/.local/share/signal-cli" --env "SIGNAL_CLI_UID=1000" --env "SIGNAL_CLI_GID=1000" --entrypoint "/entrypoint.sh" --volume "/share/CACHEDEV1_DATA/Container/container-station-data/lib/docker/volumes/app-1_signal-cli-data/_data:/home/.local/share/signal-cli" bbernhard/signal-cli-rest-api:latest
+```

docs/ARCHITECTURE.md

@@ -0,0 +1,90 @@
+# Architecture
+
+TurboBot is a Python Signal bot that routes incoming messages through a small handler framework. The runtime entrypoint is `run.py`, and feature code lives mostly in `handlers/`, `utils/`, and `tool_functions/`.
+
+## Runtime overview
+
+`run.py` configures and starts a `SignalBot` instance using environment variables such as `SIGNAL_API_URL` and `BOT_NUMBER`. The bot registers `TurboBotCommand`, whose `handle` method is the main message-processing path.
+
+Every normal bot reply is prefixed with `LOGMSG`, currently:
+
+```text
+----TURBOBOT----
+```
+
+`TurboBotCommand.handle` reads the raw Signal message, extracts useful metadata, determines whether the message is private or group-based, applies allow/ignore configuration, and then dispatches matching messages to handlers.
+
+## Authorization and routing
+
+Message routing is controlled by environment variables parsed through `utils.misc_utils.parse_env_var`:
+
+- `CONTACT_NUMBERS`: allowed private contacts. Use `true` to allow all, `false`/unset to allow none depending on call site, a single value, or a semicolon-separated list.
+- `GROUP_NAMES`: allowed group names. Use `true` to allow all, or a semicolon-separated list of group names.
+- `IGNORE_GROUPS`: optional group names that should be ignored even if otherwise allowed.
+
+Group messages are resolved from Signal internal group IDs to group metadata using `find_group_by_internal_id` in `run.py`.
+
+## Built-in command handling
+
+Before dynamic handlers are consulted, `TurboBotCommand.handle` checks a few direct commands and special cases, including:
+
+- `#ping`: health check response.
+- Reddit URLs: direct video download/reply path.
+- `#status`: machine and git status.
+- `#reboot`: exits the bot process so the surrounding launcher can restart it.
+- `#help`: lists help text from all dynamically discovered handlers.
+
+## Handler discovery
+
+Handlers are discovered by `BaseHandler.get_all_handlers()` in `handlers/base_handler.py`.
+
+Discovery rules:
+
+1. Iterate over Python modules in the `handlers/` directory.
+2. Import each module as `handlers.<module_name>`.
+3. Inspect classes in the module.
+4. Include classes that subclass `BaseHandler`.
+5. Exclude `BaseHandler` itself.
+6. Exclude classes with `is_intermediate = True`.
+
+Because discovery is dynamic, most new handlers do not need to be registered manually. Put the handler class in `handlers/`, subclass `BaseHandler` or a subclass such as `HashtagHandler`, and make sure it is not marked intermediate.
+
+## Handler execution contract
+
+The base contract is defined by `handlers/base_handler.py`:
+
+- `can_handle(self) -> bool`: return true if the handler should process the current input string.
+- `process_message(self, msg, attachments) -> dict`: return a dictionary with:
+  - `message`: text to send back to Signal.
+  - `attachments`: a list of base64-encoded attachments.
+- `get_name() -> str`: a human-readable handler name used by help output.
+- `get_help_text() -> str`: help text used by `#help`.
+
+`BaseHandler.process_message` assumes subclass implementations provide `get_message()` and `get_attachments()`. Handlers can override `process_message` when they need custom behavior.
+
+## Hashtag command abstraction
+
+`handlers/hashtag_handler.py` provides `HashtagHandler`, a convenience subclass for commands such as `#gpt`, `#mmw`, `#golf`, and `#asteroid`.
+
+A hashtag handler supplies:
+
+- `get_hashtag()`: the command pattern, such as `r"#gpt"`.
+- `get_substring_mapping()`: positional dot-argument names and defaults.
+
+For example, a message like `#gpt.gpt-4.1 explain this` can be split into the command, model substring, and cleaned prompt text.
+
+## GPT tool subsystem
+
+`handlers/gpt_handler.py` implements the `#gpt` command and dynamically loads optional function tools from `tool_functions/`.
+
+Tool modules are ordinary Python files that expose `TOOL_SPEC` and `TOOL_FN`. See `docs/GPT_TOOLS.md` for the tool-authoring contract.
+
+## Important paths
+
+- `run.py`: application entrypoint and message dispatch.
+- `handlers/base_handler.py`: common handler base class and dynamic discovery.
+- `handlers/hashtag_handler.py`: helper for hashtag/dot-argument command handlers.
+- `handlers/gpt_handler.py`: OpenAI-backed GPT and image-generation handler.
+- `tool_functions/`: dynamically loaded GPT function tools.
+- `utils/`: shared helpers for env parsing, media conversion, Reddit, video scraping, machine info, and git info.
+- `tests/`: unittest-based test suite with Signal API mocks.

docs/CONFIGURATION.md

@@ -0,0 +1,70 @@
+# Configuration
+
+TurboBot is configured mostly through environment variables loaded by `run.sh` before `run.py` starts.
+
+## Secret file
+
+`run.sh` currently looks for a file named `secret.txt` in the repository root and sources it as shell code.
+
+Create a local `secret.txt` from the checked-in template:
+
+```bash
+cp secret.example.txt secret.txt
+```
+
+Then edit `secret.txt` with real values. Do not commit real secrets.
+
+Older project notes referred to `secrets.txt` or `secrets.sh`; the current launcher reads `secret.txt`.
+
+## Environment variables
+
+### Required for normal operation
+
+```bash
+export SIGNAL_API_URL="signal-cli:8181"
+export BOT_NUMBER="+15555555555"
+```
+
+- `SIGNAL_API_URL`: URL or host:port for the signal-cli REST API.
+- `BOT_NUMBER`: registered Signal phone number for the bot.
+
+### Message allow/ignore controls
+
+```bash
+export CONTACT_NUMBERS="+15555555555;+15555555556"
+export GROUP_NAMES="My Signal Group;Another Signal Group"
+export IGNORE_GROUPS="TurboBot Devel"
+```
+
+- `CONTACT_NUMBERS`: private contacts allowed to use the bot.
+- `GROUP_NAMES`: Signal groups allowed to use the bot.
+- `IGNORE_GROUPS`: optional group names to ignore even if otherwise allowed.
+
+`CONTACT_NUMBERS`, `GROUP_NAMES`, and `IGNORE_GROUPS` are parsed by `utils.misc_utils.parse_env_var`:
+
+- unset or empty values become `None`.
+- exact lowercase `true` and `false` become booleans.
+- semicolon-separated values become lists.
+- a single non-empty value becomes a one-item list.
+
+### Optional feature credentials
+
+```bash
+export OPENAI_API_KEY="replace-with-your-openai-api-key"
+export INSTA_USERNAME="myuser"
+export INSTA_PASSWORD="replace-with-your-instagram-password"
+```
+
+- `OPENAI_API_KEY`: enables the `#gpt` handler and OpenAI model/tool calls.
+- `INSTA_USERNAME` and `INSTA_PASSWORD`: used by Instagram-related functionality when enabled.
+
+## Docker Compose configuration
+
+`docker-compose.yml` also sets deployment variables for the bot containers:
+
+- `GIT_REPO_URL`: repository to clone/fetch.
+- `GIT_REPO_PATH`: path inside the container where the repository is stored.
+- `GIT_REPO_BRANCH`: branch to reset to and run.
+- `SETUP_SCRIPT_NAME`: bootstrap script name, normally `setup.sh`.
+
+See `docs/OPERATIONS.md` for the full container startup flow.

docs/GPT_TOOLS.md

@@ -0,0 +1,71 @@
+# GPT function tools
+
+The `#gpt` handler can expose local Python functions as model-callable tools. Tool modules live in `tool_functions/` and are loaded dynamically by `handlers/gpt_handler.py`.
+
+## Loader rules
+
+`load_function_tools()` applies these rules:
+
+1. Look in `tool_functions/`.
+2. Load files ending in `.py`.
+3. Skip files whose names start with `_`.
+4. Import each module.
+5. Read `TOOL_SPEC` and `TOOL_FN` from the module.
+6. Use `TOOL_SPEC["name"]` as the tool name.
+7. Store `TOOL_FN` as the function to execute when the model calls that tool.
+
+Modules missing `TOOL_SPEC`, `TOOL_FN`, or a tool name are skipped with a warning.
+
+## Tool function contract
+
+A tool function should:
+
+- Accept keyword arguments described by `TOOL_SPEC["parameters"]`.
+- Validate inputs and raise clear exceptions for invalid data.
+- Return a dictionary when possible.
+- Include text output under a key such as `text`.
+- Include base64 attachments under `attachments` when generating files or images.
+
+Existing examples:
+
+- `tool_functions/coin_flip.py`: simple text result.
+- `tool_functions/plot_from_data.py`: generates a plot image attachment.
+
+## Minimal example
+
+```python
+from typing import Any, Dict
+
+
+def add_numbers(a: float, b: float) -> Dict[str, Any]:
+    return {"text": f"{a} + {b} = {a + b}", "attachments": []}
+
+
+TOOL_SPEC: Dict[str, Any] = {
+    "type": "function",
+    "name": "add_numbers",
+    "description": "Add two numbers and return the sum.",
+    "parameters": {
+        "type": "object",
+        "additionalProperties": False,
+        "properties": {
+            "a": {"type": "number"},
+            "b": {"type": "number"},
+        },
+        "required": ["a", "b"],
+    },
+}
+
+TOOL_FN = add_numbers
+```
+
+## Attachment behavior
+
+When a tool returns base64-encoded attachments, `handlers/gpt_handler.py` collects them and includes them in the Signal response. Keep attachments small enough for Signal to send reliably.
+
+## Safety notes
+
+- Do not expose tools that can run arbitrary shell commands from model-provided arguments.
+- Validate URLs, filenames, and numeric ranges.
+- Avoid writing secrets or sensitive local files into tool responses.
+- Prefer deterministic tools with narrow parameter schemas.