go-datamass (Go Data API)

High-performance REST API for DataMass people/company search, per-user API keys, and request logging. Consumed by the Laravel app (datamass) via App\Services\DataApiService.

---

Architecture

Laravel (nginx / php-fpm)
    │  DATA_API_URL  (e.g. http://127.0.0.1:8080)
    │  master key  →  POST/PATCH/DELETE /api/v1/keys
    │  user dm_…   →  /people, /companies, /logs
    ▼
Go API  (:8080 by default)
    │
    ├── PostgreSQL / Aurora
    │     member_data_staging   (people)
    │     firmographic_data     (companies)
    │     api_tokens            (key validation authority)
    │     api_request_logs      (usage for Laravel Logs UI)
    └── Redis (optional cache; app runs without it)

Credential	Env var (Go)	Used for
Master key	MASTER_KEY	Key provision / revoke / disable only
User token	stored in api_tokens	All data + log endpoints

MASTER_KEY must match Laravel DATA_API_MASTER_KEY.

Companion repo: datamass.

---

Prerequisites

• Go 1.22+ — https://go.dev/dl/
• PostgreSQL (local or Aurora) with network access from this host
• Redis optional (REDIS_ADDR); if unreachable, caching is skipped
• psql (or any SQL client) to run migrations / load CSVs

---

Fresh clone — first-time setup

git clone <go-datamass-repo-url> go-datamass
cd go-datamass

cp env.example .env

1. Edit .env

APP_ENV=development
PORT=8080

DB_HOST=127.0.0.1          # or Aurora reader/writer endpoint
DB_PORT=5432
DB_NAME=people               # your database name
DB_USER=api_user
DB_PASSWORD=secret
DB_SSLMODE=disable           # use "require" for Aurora

REDIS_ADDR=127.0.0.1:6379

# Shared with Laravel DATA_API_MASTER_KEY — pick a long random string in prod
MASTER_KEY=dev-master-change-me

2. Create tables / indexes

Run against the same Postgres the API will use:

# Example local:
psql "postgres://api_user:secret@127.0.0.1:5432/people?sslmode=disable" \
  -f database/migrations.sql

That script creates (among other things):

• member_data_staging, firmographic_data (+ indexes)
• api_tokens, api_request_logs

Some CREATE INDEX CONCURRENTLY statements cannot run inside a transaction. If your client wraps the file in a transaction, run those statements one at a time.

3. Load sample (or production) data

Sample CSVs ship in the repo root:

# People
psql … -c "\copy member_data_staging FROM 'member_data_staging.csv' WITH (FORMAT csv, HEADER true, DELIMITER ',', QUOTE '\"')"

# Companies
psql … -c "\copy firmographic_data FROM 'firmographic_data.csv' WITH (FORMAT csv, HEADER true, DELIMITER ',', QUOTE '\"')"

On Aurora you may COPY from an S3/EC2 path instead — same column layout, all TEXT.

4. Install modules & run

go mod tidy
go run main.go

You should see something like:

✓ Connected to Aurora PostgreSQL at …
✓ Connected to Redis at …        # or a warning if Redis is down
🚀 Server running on :8080

5. Health check

curl -s http://127.0.0.1:8080/health
# {"status":"healthy"}

---

How to run the Go API (day to day)

Development

cd go-datamass
go run main.go

Or rebuild after pulls:

go build -o datamass-api .
./datamass-api

Production (recommended: systemd on the app server)

Keep Go bound to localhost; Laravel on the same box calls http://127.0.0.1:8080.

# Build a static binary
cd /opt/go-datamass
git pull
go mod tidy
CGO_ENABLED=0 go build -o /usr/local/bin/datamass-api .

# Example unit: /etc/systemd/system/datamass-api.service

[Unit]
Description=DataMass Go API
After=network.target postgresql.service

[Service]
Type=simple
User=www-data
WorkingDirectory=/opt/go-datamass
EnvironmentFile=/opt/go-datamass/.env
ExecStart=/usr/local/bin/datamass-api
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target

sudo systemctl daemon-reload
sudo systemctl enable --now datamass-api
sudo systemctl status datamass-api
journalctl -u datamass-api -f

Optional: nginx reverse proxy (public API)

Only needed if third parties call Go directly (not via Laravel). Example:

server {
    listen 443 ssl;
    server_name api.your-domain.example;

    location / {
        proxy_pass http://127.0.0.1:8080;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }
}

For the normal Laravel integration, skip this and leave Go on 127.0.0.1:8080.

---

Wire Laravel to this API

On the Laravel server .env:

DATA_API_URL=http://127.0.0.1:8080
DATA_API_MASTER_KEY=dev-master-change-me   # identical to MASTER_KEY here
DATA_API_TIMEOUT=10
DATA_API_CHUNK_SIZE=500

Then:

1. Start / restart Go (systemctl restart datamass-api or go run main.go).
2. Confirm curl http://127.0.0.1:8080/health.
3. In Laravel, register a user (or re-provision keys — see Laravel README).
4. Hit API Management → Logs and Keys in the UI.

Flow for keys:

1. Laravel creates dm_… in its api_keys table.
2. Laravel calls POST /api/v1/keys with Authorization: Bearer <MASTER_KEY>.
3. Go inserts into api_tokens.
4. Later UI/export requests send Authorization: Bearer <dm_…>; Go validates against api_tokens.

---

Endpoints (quick reference)

Paginated envelope: { "data": [...], "total", "page", "page_size" }

Method	Path	Auth
GET	/health	none
POST	/api/v1/keys	master
PATCH	/api/v1/keys/:token	master (is_active, label)
DELETE	/api/v1/keys/:token	master
GET	/api/v1/people	user token
GET	/api/v1/people/count	user token
GET	/api/v1/people/by-work-email?email=	user token
GET	/api/v1/people/by-personal-email?email=	user token
GET	/api/v1/companies	user token
GET	/api/v1/companies/count	user token
GET	/api/v1/logs	user token
GET	/api/v1/logs/stats	user token

User-token requests should also send X-Account-ID: <laravel_user_id> (Laravel’s DataApiService does this automatically).

Manual smoke tests

# Health
curl -s http://127.0.0.1:8080/health

# Provision a key (master)
curl -s -X POST http://127.0.0.1:8080/api/v1/keys \
  -H "Authorization: Bearer dev-master-change-me" \
  -H "Content-Type: application/json" \
  -d '{"user_id":1,"token":"dm_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa","slot":1,"label":"System key"}'

# Search people (user token)
curl -s "http://127.0.0.1:8080/api/v1/people?page=1&page_size=5" \
  -H "Authorization: Bearer dm_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa" \
  -H "X-Account-ID: 1"

# Revoke
curl -s -X DELETE \
  "http://127.0.0.1:8080/api/v1/keys/dm_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa" \
  -H "Authorization: Bearer dev-master-change-me"

---

Data tables

People — member_data_staging

All columns TEXT. Sample: member_data_staging.csv.

Companies — firmographic_data

All columns TEXT. Sample: firmographic_data.csv.

API JSON fields map from source columns (e.g. web_domain → domain, international_label → industry, employees_total → employee_count).

---

Troubleshooting

Symptom	Fix
Failed to connect to database	Check DB_*, security groups, DB_SSLMODE
Redis warning on startup	OK — API works without cache
Laravel provision returns 401	MASTER_KEY ≠ DATA_API_MASTER_KEY
Data requests 401	Token missing/revoked in api_tokens
Empty search results	Staging tables not loaded / wrong DB
Port in use	Change PORT in .env and Laravel DATA_API_URL

# Is anything listening?
ss -lntp | grep 8080   # or: netstat -an | findstr 8080   (Windows)

---

Project layout

go-datamass/
├── main.go
├── env.example
├── database/migrations.sql
├── config/
├── models/
├── store/          # api_tokens + request logs
├── handlers/       # people, companies, keys, logs
├── middleware/     # auth, rate limits, request logger
├── cache/
├── router/
└── member_data_staging.csv / firmographic_data.csv   # samples