Corncrake

A configurable REST API for statutory survey submissions to national statistical institutes.

Ships with a built-in tenant for the CSO EHECS (Earnings, Hours and Employment Costs Survey) — add your own scheme by implementing two small interfaces. Modelled on Revenue Ireland's PAYE Modernisation REST API architecture.

This is an independent open-source project. It is not affiliated with or endorsed by the Central Statistics Office, Ireland.

Part of the Corncrake toolchain.

---

Why Corncrake

The existing EHECS submission process requires generating a schema-specific XML file and uploading it manually through a web portal. There is no direct API integration path — every system, regardless of how sophisticated, ends up exporting a file and clicking upload.

Corncrake provides the API layer that removes the manual step — payroll software submits directly in real time, the same way Revenue's PAYE Modernisation works. The tenant architecture means the same infrastructure serves multiple survey types without duplicating code.

---

Adding a new survey tenant

Create a package under internal/tenants/ that implements tenant.Tenant and calls tenant.Register from init():

// internal/tenants/mysurvey/mysurvey.go
package mysurvey

import (
    "encoding/json"
    "github.com/CathalByrneGit/corncrake/internal/models"
    "github.com/CathalByrneGit/corncrake/internal/tenant"
)

func init() { tenant.Register(&MySurvey{}) }

type MySurvey struct{}

func (s *MySurvey) ID() string   { return "my-survey" }
func (s *MySurvey) Name() string { return "My Statistical Survey" }

func (s *MySurvey) ValidateSchema(body json.RawMessage) []models.ValidationItem {
    // your structural checks → HTTP 400
    return nil
}

func (s *MySurvey) ValidateLogic(body json.RawMessage) (errors, warnings []models.ValidationItem) {
    // your cross-field checks → HTTP 422
    return nil, nil
}

func (s *MySurvey) ItemCount(body json.RawMessage) int {
    // how many records are in the body (used in run status summaries)
    return 0
}

Then add a blank import to main.go:

import _ "github.com/CathalByrneGit/corncrake/internal/tenants/mysurvey"

Submissions are immediately available at /corncrake/v1/my-survey/submissions/.... No other code changes needed — the auth, routing, rate limiting, token revocation, and database layer all work unchanged.

Optional: lookup data

To expose reference data at GET /corncrake/v1/my-survey/lookups/{name}, also implement tenant.LookupProvider:

func (s *MySurvey) Lookup(name string, r *http.Request) (data any, count int, ok bool) {
    switch name {
    case "codes":
        return myCodes, len(myCodes), true
    }
    return nil, 0, false
}

?search= and other query parameters are available via r.URL.Query(). Tenants that don't implement LookupProvider return 404 on any lookup request — they don't inherit another tenant's data.

---

Quick start

git clone https://github.com/CathalByrneGit/corncrake.git
cd corncrake
go run .

• Swagger UI: http://localhost:3000/api-docs
• Health check: http://localhost:3000/corncrake/v1/health

---

Endpoints

All submission and lookup routes are tenant-scoped under /{tenantID}/. The EHECS tenant ships with the service; add your own by implementing tenant.Tenant (see Adding a new survey tenant).

Method	Path	Auth	Description
GET	/corncrake/v1/health	None	Health check
POST	/corncrake/v1/{tenantID}/submissions/{holdingNumber}/{taxYear}/{quarter}/{runReference}/{submissionId}	JWT	Create submission
GET	/corncrake/v1/{tenantID}/submissions/{holdingNumber}/{taxYear}/{quarter}/{runReference}/{submissionId}	JWT	Retrieve submission
GET	/corncrake/v1/{tenantID}/submissions/{holdingNumber}/{taxYear}/{quarter}/{runReference}	JWT	Run status
GET	/corncrake/v1/{tenantID}/submissions/{holdingNumber}	JWT	List submissions
GET	/corncrake/v1/{tenantID}/lookups/{lookupName}	JWT	Tenant-specific reference data
GET	/corncrake/v1/{tenantID}/lookups/periods/{holdingNumber}	JWT	Open quarterly reporting periods

EHECS lookup names (tenant ehecs): occupation-codes (supports ?search=), schema-version.

Full specification: docs/openapi.yaml

A note on path parameters

The submission endpoint uses multiple path parameters following Revenue's PAYE Modernisation URL convention. Each segment serves as an idempotency key — submitting the same submissionId twice is safe and returns the original result without duplicate processing. This pattern is intentional and familiar to any developer already integrating with Revenue's API.

---

Authentication

Bearer JWT in the Authorization header. At minimum the token must carry:

{ "holdingNumber": "CSO123456" }

Using corncrake-cli or the web formatter? A token is issued directly to your organisation. You do not need to think about software identifiers.

Integrating a payroll software product? Also include:

{
  "holdingNumber":   "CSO123456",
  "softwareUsed":    "YourProductName",
  "softwareVersion": "2026.1"
}

Note on JWT security: JWTs are signed, not encrypted — claims in the payload are readable by anyone who holds the token. Never put sensitive data in claims. Tokens must be issued by the Corncrake auth service; clients must not construct or modify their own tokens. For machine-to-machine integration, OAuth 2.0 client credentials flow (exchanging a client_id and client_secret for a short-lived access token) is the recommended production pattern.

---

Configurable database backend

DB_DRIVER	Backend	Build tag
memory	In-memory (default)	none
postgres	PostgreSQL	-tags postgres
sqlserver	Microsoft SQL Server	-tags sqlserver

go build                    # memory only — no DB deps, works offline
go build -tags postgres     # include PostgreSQL driver
go build -tags sqlserver    # include SQL Server driver
go test ./...               # always runs against memory store

# Runtime
DB_DRIVER=sqlserver
DATABASE_URL=sqlserver://user:pass@host:1433?database=corncrake&encrypt=true

---

Validation (CSO EHECS tenant)

Source: CSO Notes for Payroll Software Providers on EHECS Requirements v4.0

HTTP 400 — Schema: PPSN format, occupation codes 1–9, employment type enum, all numeric fields ≥ 0.

HTTP 422 — Logic: overtimePay > 0 requires overtimeHours > 0 · grossEarnings >= overtimePay · basicHours ≤ 2,184 per quarter.

---

Credential storage

• Plain-text secret generated once, emailed to the employer, never stored
• PBKDF2-HMAC-SHA256 hash + unique salt persisted per credential (Argon2id target — see roadmap)
• SQL Server deployments: hash and salt columns use Always Encrypted
• Append-only audit log for every AUTH_SUCCESS and AUTH_FAIL (token verification events included)

---

Cryptography roadmap

Timeline	Action
Now	Enable hybrid post-quantum TLS at the load balancer (AWS ALB, Azure, Cloudflare — no code changes needed)
Near-term	Migrate credential hashing PBKDF2-HMAC-SHA256 → Argon2id (GPU-resistant; stdlib golang.org/x/crypto/argon2)
12–18 months	Migrate JWT signing RS256 → ML-DSA (NIST FIPS 204) once Go ecosystem has stable support; migrate auth to OAuth 2.0 client credentials
No action needed	AES-256 Always Encrypted is not meaningfully vulnerable to quantum attack

---

Project structure

corncrake/
├── main.go
├── internal/
│   ├── models/          Domain types
│   ├── validators/      Schema and logic validation
│   ├── services/        SubmissionStore interface + Memory/Postgres/SQLServer
│   ├── middleware/      JWT auth
│   └── handlers/        HTTP handlers + tests
├── docs/openapi.yaml    OpenAPI 3.0 spec
└── .env.example

---

Configuration

Variable	Default	Description
PORT	3000	Listen port
APP_ENV	development	json logging; enables HTTPS enforcement when set to production
JWT_PUBLIC_KEY	—	PEM-encoded RSA public key — enables RS256 JWT verification (recommended for production)
JWT_SECRET	dev secret	HS256 fallback — used only when JWT_PUBLIC_KEY is not set. Change in production
DB_DRIVER	memory	memory | postgres | sqlserver
DATABASE_URL	—	Required for postgres / sqlserver

---

Testing

go test ./...        # 17 tests, no database needed
go test -race ./...  # with race detector

---

Ecosystem

Repo	Description
corncrake	This service
corncrake-sdk	Go library — format and submit from any application
corncrake-cli	CLI tool — map, validate, submit, get XML from the terminal

---

Licence

MIT — Independent open-source project, not affiliated with the CSO.