Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion chain-spec/chain-spec-raw.json

Large diffs are not rendered by default.

42 changes: 42 additions & 0 deletions docs/OPERATOR_KIT.md
Original file line number Diff line number Diff line change
Expand Up @@ -55,3 +55,45 @@ Your Machine FPS Infrastructure
| `substrate_connected: false` | RPC unreachable | Check WSS connectivity to `materios.fluxpointstudios.com` |
| High finality gap (>10) | Chain stalled | Check if block production is healthy on the explorer |
| Cert not submitted | Account not funded | Ask FPS team to verify MATRA balance |

### Sync stuck at snapshot floor / peer-ban loop

Symptom: your node restored a snapshot, sees `target=#N` (the real chain tip), but stays at `best=#snapshot_floor` indefinitely. Logs show repeating lines like:
```
Report 12D3KooW...: -2147483648 to -2147483648. Reason: Same block request multiple times. Banned, disconnecting.
```
and `Idle (0 peers)` between bans.

This means your node is rejecting every peer that tries to send it new blocks. Your **state is fine** (snapshot integrity is independent of this); the issue is in the sync-protocol handshake.

**Diagnostic 1 β€” confirm canonical state match.** From a healthy reference (e.g. ask FPS for the canonical hash at your `best` block), then on your node:
```bash
curl -s -H "Content-Type: application/json" \
-d '{"jsonrpc":"2.0","method":"chain_getBlockHash","params":[<your-best-block-number>],"id":1}' \
http://127.0.0.1:9944
```
If your hash matches canonical, the snapshot is fine and the issue is sync-protocol (proceed to D2). If it differs, your snapshot is from a forked chain β€” request a fresh one.

**Diagnostic 2 β€” isolate sync to a single trusted peer.** Stop the node, edit `/etc/systemd/system/materios-node-spo.service` (the `ExecStart` line), and add:

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Point operators at the Compose service, not systemd

For operators who used the recommended bootstrap/operator-kit path documented above, the installer writes an $OPERATOR_DIR/docker-compose.yml and runs the node as a Docker Compose service; it does not create /etc/systemd/system/materios-node-spo.service. In that environment this troubleshooting step sends them to edit a non-existent unit, so the reserved-node workaround will not be applied when they hit the peer-ban loop. Please direct these users to update the generated Compose command and restart materios-node with Docker Compose, or gate the systemd instructions to deployments that actually use that unit.

Useful? React with πŸ‘Β / πŸ‘Ž.

```
--reserved-nodes /dns4/bootnode.materios.fluxpointstudios.com/tcp/30333/p2p/12D3KooWPueKoxRAirTTKH4Y2qQAsJDegWMjS4k89Z7izCbZKgkM \
--reserved-only \
--in-peers 50 --out-peers 25
```
Then `systemctl daemon-reload && systemctl restart materios-node-spo.service`.

This forces your node to peer ONLY with the FPS Gemtek validator (bypasses libp2p DHT discovery) and increases peer-slot capacity. If sync progresses past the floor with `--reserved-only`, the bad peer was somewhere in the network and DHT discovery was finding it. Once you're caught up to tip, drop `--reserved-only` so you can serve other peers.

**Diagnostic 3 β€” verbose sync logs.** Add to the same `ExecStart`:
```
&& RUST_LOG=sync=debug,sub-libp2p=info exec /usr/local/bin/materios-node-spo ...
```
(prepend `RUST_LOG=...` before the `exec` line in the bash invocation). Tail `node-spo.log` and look at the lines just before a "Banned" event. The debug logs will show the exact BlockRequest IDs and timing β€” if the same request_id arrives twice from the same peer in <1s, you'll see it.

**Common root causes:**

- **Stale libp2p peer in the DHT.** A node that's no longer running but is still advertised. `--reserved-only` is the immediate workaround.
- **Slow inherent-data verification.** If your `cardano-db-sync` postgres is on a slow disk, partner-chain inherent-data lookups during block import can take 1-4s. Peers retry, and your sync layer flags the retry as duplicate. Move postgres to a faster disk (NVMe), or increase postgres's `shared_buffers` and `work_mem`.
- **Binary version skew.** Your `materios-node` binary may be older than the network's current version. Re-run `bootstrap-validator.sh` to pull the latest binary from `/releases/`.

If none of these help, capture the verbose log around 5 ban events and share with FPS.