---

sidebar_position: 1

---

🐳 SCS

ITPal Service Control Standard

1. 📋 Overview

The ITPal Service Control Standard (SCS) defines a single, scriptable, and observable way to manage services across Linux servers. It is the foundation of ITPal’s infrastructure operations, built around:

• 🐳 Docker-first architecture
• 📜 Script-based service control (adminctl)
• 📊 Matrix-based monitoring and operator feedback
• 🔄 Consistency, automation, and traceability

With SCS, infrastructure isn’t just about deploying containers — it’s about running them in a predictable, monitored, and upgradeable way through clear protocols and tools.

🎯 Goals

• Define 🏗️ a standard layout, naming rules, and control scripts for all services.
• Allow fully 🌐 remote and automated service operations through the CLI and Matrix bot.
• Reduce manual steps while increasing 🔍 visibility and 📈 monitoring.
• Establish 🔄 clear lifecycle states for every service (start, stop, restart, build, backup, restore, etc).

📏 Scope

SCS applies to:

• All backend services and applications managed by ITPal, especially those running in Docker.

• Shell scripts that perform service operations (start, deploy, logs, etc.).

• Communication between services and:

  • adminctl (the privileged host command dispatcher),
  • The 🤖 Matrix Operator Bot (for remote commands, status, and alerts).

2. ⚙️ Installation

2.1 📦 Installing SCS CLI

The SCS CLI is installed using the provided script. It installs globally:

/etc/control/scs/install.sh

After installation, the CLI is available as scs.

2.2 🎮 Using SCS CLI

Basic commands:

# Help and available commands
scs help

# List all services
scs list

# Start a service
scs start <service-name>

# Stop a service
scs stop <service-name>

# Check service status
scs status <service-name>

# View logs
scs logs <service-name>

---

3. 🎛️ Control CLI

The cli.sh script is the central interface for all services under SCS. It hides Docker commands, service dependencies, build arguments, lifecycle hooks, and Git operations behind a consistent Bash interface.

This CLI is used both by humans (via SSH or shell) and by higher-level agents like adminctl and the Matrix Operator Bot.

3.1 🏷️ Service Declarations

Services are declared in config.sh using associative arrays:

• containers — service → container name.
• images — service → Docker image tag.
• services — which services are enabled.
• services_path — where each service lives.
• run_args — Docker run arguments.
• repos_path, github_repos — Git repos for version control.

This serves as the manifest of services, loaded dynamically by cli.sh.

Unlike YAML tools (like Docker Compose), this Bash format supports variables, conditions, and custom repo tracking.

3.2 ⚡ Actions

cli.sh includes built-in actions:

• 🔨 build: Build Docker image.
• ▶️ start: Start service (with dependencies).
• ⏹️ stop: Stop container.
• 🔄 restart: Restart service.
• 📊 status: Show Docker container status.
• 📋 logs: Follow logs.
• 🐚 shell: Open a shell inside container/image.
• ⚙️ exec: Run arbitrary commands in container.
• 🏷️ version: Print Git metadata.
• 🔍 check-update: Compare repo with GitHub.
• ⬆️ upgrade: Update repo, rebuild, restart.
• 💾 backup: Create a backup.
• 📥 restore: Restore from backup.
• ⏰ setup-backup: Setup scheduled backups.
• 📋 list-backups: Show backups.

3.3 🪝 Hooks and Conventions

Services can define lifecycle scripts in their directories:

• start.sh, stop.sh, restart.sh — override defaults.
• pre-start.sh, post-start.sh, pre-restart.sh, post-restart.sh — hooks.
• upgrade.sh, check-update.sh, version.sh, after-upgrade.sh — custom logic.

This makes behavior customizable without touching the CLI core.

3.4 📁 Service Folder Structure

Typical structure under /etc/<service>/:

/etc/<service>/
├── .env
├── .env.defaults
├── adminctlConfig.js
├── repo/
├── start.sh
├── stop.sh
├── restart.sh
├── pre-start.sh
├── post-start.sh
├── post-stop.sh
├── build.sh
├── version.sh
├── check-update.sh
├── upgrade.sh
├── backup.sh
├── restore.sh
├── Dockerfile

All files are optional. Defaults assume a single container run with docker run and run_args.

3.5 🎨 CLI Design Principles

• Config is declarative (config.sh).
• 🪝 Hooks take priority if defined.
• 🔗 Dependencies respected with start_with_deps.
• Docker network auto-created.
• Consistent 📋 log output with emoji markers.
• shell and exec reuse run_args.

3.6 💻 Usage Patterns

scs start synapse       # Start synapse + dependencies
scs restart kuma        # Restart kuma
scs check-update all    # Git status for all services
scs upgrade mas         # Update MAS
scs status all          # Show status for all services
scs status webhook      # Show status for webhook service
scs logs webhook        # Follow logs
scs exec hookshot "ps aux"
scs backup authentik    # Backup authentik
scs restore synapse     # Restore synapse
scs setup-backup kuma   # Setup backups for kuma

These can be run directly, over SSH, via adminctl, or through automation.

⚙️ Why Not Kubernetes? SCS is designed for developers/operators who want:

• Running the latest dev commits
• Easy forking/patching upstream
• Full control over Docker args and repo tracking

Kubernetes focuses on scaling and abstraction. SCS prioritizes speed, flexibility, and full control.

---

4. 💾 Backup and Restore

SCS includes backup and restore for services, with scheduling, S3 integration, and retention.

4.1 📋 Overview

• ⏰ Scheduled backups (systemd timers).
• ☁️ S3 integration.
• 🗑️ Retention policies.
• 🔧 Service-specific scripts.
• 🎛️ Unified CLI interface.

4.2 ⚙️ Configuration

In config.sh:

declare -A backup_services=(
  ["authentik"]="daily:30"
  ["synapse"]="weekly:90"
  ["kuma"]="monthly:365"
)

BACKUP_S3_BUCKET="your-bucket"
BACKUP_S3_REGION="us-east-1"
BACKUP_AWS_PROFILE="backup-profile"

• daily → every midnight
• weekly → Sunday midnight
• monthly → first Sunday
• Retention → days before deletion

4.3 🎮 CLI Commands

scs setup-backup <service>   # Setup automated backups
scs backup <service>         # Run manual backup
scs list-backups <service>   # List backups
scs restore <service> <file> # Restore backup

4.4 🔧 Service Backup Scripts

Each service provides backup.sh:

1. Create archive.
2. Place in backups/.
3. Exit code 0 = success.

4.5 🔄 Restore Scripts

Each service provides restore.sh:

1. Accept archive path.
2. Extract/validate.
3. Restore service.
4. Exit code 0 = success.

4.6 ☁️ S3 Integration

If S3 configured:

• Upload backups after creation.
• Apply retention policies.
• Restore pulls from S3.
• Local files cleaned up.

Path structure:

s3://<bucket>/<service>/<backup>.tar.gz

4.7 🔄 Lifecycle

1. Systemd triggers backup.
2. Service backup.sh runs.
3. Archive uploaded to S3.
4. Old backups removed.
5. Service restarted if needed.

Debugging:

systemctl status scs-backup-<service>
journalctl -u scs-backup-<service>
scs backup <service>
aws s3 ls s3://<bucket>/<service>/ --profile <profile>

---

5. 🤖 adminctl

5.1 📝 adminctlConfig.js

Defines how adminctl and the Matrix Operator Bot interact with a service.

Type Definition

type AdminctlServiceConfig = {
  [serviceKey: string]: {
    serviceName: string,
    actions: string[],
    repoPath: string,
    owner: string,
    repo: string,
    branch: string,
    showTagMessage?: boolean,
    showReleaseNotes?: boolean,
    patchBranches?: string[],
    upstream?: { owner: string, repo: string },
    filesToCheck?: string[],
    getLocalVersion?: Function,
    upgrade?: Function,
    afterUpgrade?: Function,
  },
};

🔑 Key Descriptions

• 🏷️ serviceName: Name of service.
• ⚡ actions: Supported commands.
• 📁 repoPath: Local repo path.
• 👤 owner: GitHub owner.
• 📦 repo: Repo name.
• 🌿 branch: Branch tracked.
• 🏷️ showTagMessage: Include tag message.
• 📋 showReleaseNotes: Include release notes.
• 🔧 patchBranches: Extra patch branches.
• ⬆️ upstream: Override fork upstream.
• 📄 filesToCheck: Files to diff.
• 🏷️ getLocalVersion: Custom version logic.
• ⬆️ upgrade: Custom upgrade function.
• 🔄 afterUpgrade: Logic after upgrade.