feat(cluster): sizing report for OS, kernel, virtualization

[gsanchietti]

Summary

Adds a leader-only send-sizing-report script and timer that build an
anonymized per-node sizing report for the last complete UTC day — hardware
descriptors, Prometheus utilization percentiles, per-module workload counters
from get-facts — and ship it to the insights server. Missed days are
backfilled, up to a week back.

The report never carries identifying strings: no FQDN, IP address, host name or
DMI serial. Only numeric workload values and coarse descriptors (CPU model, OS
id, kernel release, virtualization) are forwarded.

The endpoint is read only from the cluster agent environment. With
INSIGHTS_SERVER_URL unset the timer runs and exits without sending anything,
so merging this changes nothing on a cluster that has not opted in.

Docs: docs/core/subscription.md, section Sizing report.

Configuration

Two variables in the cluster agent environment:

Variable	Meaning
INSIGHTS_SERVER_URL	Base URL of the insights server. The report is POSTed to $INSIGHTS_SERVER_URL/v1/sizing-reports with HTTP Basic auth, using system_id/auth_token from cluster/subscription. Empty or unset ⇒ nothing is sent
INSIGHTS_VERIFY_TLS	0 skips TLS certificate verification; any other value, including unset, keeps it on

How to test

1. Deploy manually on a leader node

No core update needed — one script and two units, nothing existing is touched:

#!/bin/bash
set -e
BASE=https://raw.githubusercontent.com/NethServer/ns8-core/insights/core/imageroot

curl -fsSL -o /var/lib/nethserver/cluster/bin/send-sizing-report \
    "$BASE/var/lib/nethserver/cluster/bin/send-sizing-report"
chmod 755 /var/lib/nethserver/cluster/bin/send-sizing-report

curl -fsSL -o /etc/systemd/system/send-sizing-report.service \
    "$BASE/etc/systemd/system/send-sizing-report.service"
curl -fsSL -o /etc/systemd/system/send-sizing-report.timer \
    "$BASE/etc/systemd/system/send-sizing-report.timer"

systemctl daemon-reload
systemctl enable --now send-sizing-report.timer

Or, via a full core update:

update-core --repository ghcr.io/nethserver/core --tag insights

2. Preview the payload, without sending anything

runagent -m cluster send-sizing-report --print | jq

A specific day, or a list of days, in ISO format:

runagent -m cluster send-sizing-report --print 2026-09-01 2026-09-02

Check the output carries no FQDN, IP address, host name or serial.

3. Point it at an insights server

INSIGHTS_SERVER_URL=https://controller.gs.nethserver.net/insights runagent send-sizing-report
# or
echo INSIGHTS_SERVER_URL=https://insights.example.org >> /var/lib/nethserver/cluster/state/environment
# self-signed certificate on the server side:
echo INSIGHTS_VERIFY_TLS=0 >> /var/lib/nethserver/cluster/state/environment

No restart needed — runagent reads the file at every run.

4. Send

Run the unit the timer would run:

systemctl start send-sizing-report.service
journalctl -u send-sizing-report.service -n 20 --no-pager

Expected on success:

sizing report accepted for 2026-09-02: {"accepted":true,"stored_days":1,"stored_nodes":1,"dropped":{...}}

Or send a chosen day on demand, bypassing the marker:

runagent -m cluster send-sizing-report 2026-09-01

5. Backfill and redelivery

The last accepted day lives in cluster/sizing_report. Only complete UTC days
newer than it are sent, at most 7 per run:

redis-cli HGET cluster/sizing_report last_acked_day

Rewind it to force a resend of the last three days, then run the unit again —
the marker should come back to yesterday:

redis-cli HSET cluster/sizing_report last_acked_day $(date -u -d '4 days ago' +%F)
systemctl start send-sizing-report.service
redis-cli HGET cluster/sizing_report last_acked_day

Redelivery is safe: a day is an absolute fact and the server recomputes the
stored row instead of accumulating into it.

6. A wrong day cannot poison the marker

The server answers 202 for a well-formed report even when it stores nothing
from it. Sending today — still accumulating, always outside the accepted window
— must be reported and must leave last_acked_day alone:

redis-cli HGET cluster/sizing_report last_acked_day
runagent -m cluster send-sizing-report $(date -u +%F)
redis-cli HGET cluster/sizing_report last_acked_day   # unchanged

Expected:

<4>sizing report: the server dropped 1 of the 1 day(s) sent (2026-09-03), they fall outside its accepted window
<4>sizing report stored nothing for 2026-09-03, the acknowledged day marker is left untouched

Same for a day older than the server's retention. In a mixed batch the marker
advances only to the newest day that was actually stored.

7. A run that sends nothing says why

Each no-op path logs a notice instead of exiting 0 in silence:

sizing report: nothing to send, every complete day up to 2026-09-02 is already acknowledged
sizing report: not the cluster leader, nothing to do
sizing report: cluster/subscription is not configured, nothing to do

8. Opt-out is the default

With INSIGHTS_SERVER_URL removed from
/var/lib/nethserver/cluster/state/environment, systemctl start send-sizing-report.service must succeed and leave last_acked_day untouched.

Verified on a live cluster

Single-node NS8 leader, Rocky 9.8, metrics1 + loki1 installed, insights
server on 127.0.0.1:19595:

• --print output byte-identical before/after the dead-code removal, for the
  default day and for a three-day backfill
• post_report retry matrix (network error exhaust/recover, 503 retry,
  persistent 503, 400 not retried, retries=0, verify_tls 0/1): 8/8 identical
  before and after
• endpoint unset while module/loki1/environment INSIGHTS_SERVER_URL is still
  set ⇒ nothing sent, marker unchanged — the old loki fallback is gone
• endpoint set ⇒ 202, server reports
  accepted_nodes:1 accepted_modules:6 accepted_metrics:7, all dropped_*:0
• marker rewound 3 days ⇒ 3 days delivered, marker back to yesterday
• sending today or a day older than retention ⇒ server reports dropped_day:1,
  stored_days:0, marker held; mixed batch ⇒ marker advances only to the
  newest stored day
• 23 unit assertions on parse_ack / acked_day / send_report marker
  handling (accept, reject, mixed, unparsable body, 400, 500) and 6 on the
  three main() no-op notices: all pass