feat(stats): add daily Bitcoin merge-mining metrics snapshot by nicov-iov · Pull Request #258 · rsksmart/rsk-explorer-api

nicov-iov · 2026-06-25T14:31:36Z

What

Adds a daily snapshot of Bitcoin merge-mining security metrics, persisted by the indexer for the read API (explorer-api) to expose on /api/v3/stats. Companion PR: rsksmart/explorer-api#256 (same branch name).

Metrics persisted to the new btc_merge_mining_stats table (one row per UTC day):

bitcoin_hashrate — estimated Bitcoin network hashrate
merge_mining_percentage — share of sampled BTC blocks carrying the RSK tag
rootstock_secured_hashrate — bitcoin_hashrate × merge_mining_percentage
bitcoin_blocks_sampled, merge_mined_blocks — sampling transparency

How

Detection (src/lib/rskMergeMiningTag.js): scans each coinbase's OP_RETURN outputs and input scriptSig for the RSKBLOCK: tag (RSKIP-110). Classifies by tag presence alone, never by the presence of RSK transactions. Handles both Esplora and Bitcoin Core field shapes.
Data source (src/lib/btcMempoolClient.js): mempool.space REST API — no dedicated Bitcoin node required. Treated as untrusted input: bounded timeout, retries on 429/5xx only, throttle between calls, and block-hash validation before any path interpolation. Base URL is configurable (src/lib/defaultConfig.js → bitcoin), so testnet or a self-hosted Esplora mirror is a one-line override.
Computation (src/lib/getMergeMiningStats.js): fetches the network hashrate first (fail fast), walks the latest blocksSample (1000 ≈ 7 days) blocks, tolerates per-block fetch failures, and aborts only if coverage drops below minCoverage (default 0.9) so a failing provider can't skew the ratio.
Schedule (src/lib/cronJobs.js): a new daily cron (03:20 UTC) computes and upserts the snapshot keyed by date (idempotent). On any failure it logs and leaves the previous snapshot in place.
Schema: table added to prisma/rsk-explorer-database.sql (DDL source of truth) and mirrored in prisma/schema.prisma.

Production database

This repo owns DDL for the shared database. Run once against existing production databases before deploying (the read API depends on this table):

CREATE TABLE IF NOT EXISTS btc_merge_mining_stats (
    date DATE PRIMARY KEY,
    bitcoin_hashrate VARCHAR NOT NULL,
    rootstock_secured_hashrate VARCHAR NOT NULL,
    merge_mining_percentage NUMERIC NOT NULL,
    bitcoin_blocks_sampled INT NOT NULL,
    merge_mined_blocks INT NOT NULL,
    created_at TIMESTAMPTZ NOT NULL DEFAULT now()
);

Testing

Unit tests for tag detection (Esplora + Core shapes, casing, malformed input), the computation (ratio/secured-hashrate math, genesis cap, failure-skipping, coverage floor, fail-fast hashrate), and the client (retry on 5xx, fail-fast on 4xx, hash validation, coinbase extraction). npx mocha test/rskMergeMiningTag.spec.js test/getMergeMiningStats.spec.js test/btcMempoolClient.spec.js → 15 passing. npm run lint clean.

Follow-up

The cron fails soft, so a sustained provider outage leaves the snapshot stale silently. A Prometheus freshness gauge (age of last successful snapshot) is a sensible follow-up to alert on staleness.

Configuration (prod)

Set the Bitcoin endpoint per deployment in config.json (see config-example.json, now includes a bitcoin block). The key that must match the network is bitcoin.mempoolApiUrl:

Mainnet: https://mempool.space/api (default)
Testnet: https://mempool.space/testnet/api

All bitcoin.* defaults live in src/lib/defaultConfig.js (sample size, hashrate period, minCoverage, timeouts/retries) — override in config.json only as needed.

Minor doc debt (not addressed here): the config block in README.md predates this change and doesn't list the bitcoin section, and config-example.json's db block is stale from the old stack (server/port 27017). Both are pre-existing and out of scope for this PR.

Add a daily cron job that samples the latest Bitcoin blocks, detects the RSKBLOCK: merge-mining tag in each coinbase, and persists a snapshot (bitcoin_hashrate, rootstock_secured_hashrate, merge_mining_percentage, bitcoin_blocks_sampled, merge_mined_blocks) to btc_merge_mining_stats. - RSK tag detection scans the coinbase OP_RETURN outputs and input scriptSig, classifying by tag presence alone. - Bitcoin data comes from the mempool.space REST API (no dedicated node); the client treats it as untrusted: bounded timeout, retries on 429/5xx only, throttling, and block-hash validation before path interpolation. - The walk tolerates per-block fetch failures and aborts only if coverage drops below a configurable floor, so the ratio is never skewed by a failing provider.

github-actions · 2026-06-25T14:31:46Z

Dependency Review

The following issues were found:

✅ 0 vulnerable package(s)
✅ 0 package(s) with incompatible licenses
✅ 0 package(s) with invalid SPDX license definitions
⚠️ 1 package(s) with unknown licenses.

See the Details below.

License Issues

package.json

Package	Version	License	Issue Type
axios	^1.8.4	Null	Unknown License

OpenSSF Scorecard

Package	Version	Score	Details
npm/axios	^1.8.4	Unknown	Unknown

Scanned Files

package.json

Copilot

Pull request overview

Adds a daily persisted snapshot of Bitcoin merge-mining security metrics so the read API can expose them via /api/v3/stats, backed by a new btc_merge_mining_stats table and a scheduled indexer job.

Changes:

Introduces merge-mining tag detection (RSKBLOCK:) and merge-mining stats computation over a sampled BTC block window.
Adds an Esplora-compatible mempool.space client with retries/timeouts/throttling and wires it into a new daily cron upsert.
Extends the DB schema (Prisma + SQL DDL) to store daily merge-mining metrics.

Reviewed changes

Copilot reviewed 10 out of 10 changed files in this pull request and generated 6 comments.

Show a summary per file

File	Description
test/rskMergeMiningTag.spec.js	Unit tests for RSK merge-mining tag detection (Esplora + Core shapes).
test/getMergeMiningStats.spec.js	Unit tests for sampling/coverage math and fail-fast behavior.
test/btcMempoolClient.spec.js	Unit tests for retries, non-retryable failures, and input validation.
src/lib/rskMergeMiningTag.js	Implements tag detection across coinbase input/output script hex.
src/lib/getMergeMiningStats.js	Computes sampled merge-mining percentage and secured hashrate.
src/lib/defaultConfig.js	Adds `bitcoin` config block for mempool API + sampling parameters.
src/lib/cronJobs.js	Adds a daily cron job to compute and upsert the snapshot.
src/lib/btcMempoolClient.js	Adds a mempool.space REST client with bounded timeout and retries.
prisma/schema.prisma	Adds Prisma model for `btc_merge_mining_stats`.
prisma/rsk-explorer-database.sql	Adds DDL for the new `btc_merge_mining_stats` table.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Reject non-positive sampleSize / out-of-range minCoverage so the ratio can never divide by zero and persist NaN. Use the correct OP_RETURN push opcode (0x29 = RSKBLOCK: + 32 bytes) in the test fixtures.

global.fetch is unavailable on the Node 16 deploy runtime, so the client would throw at runtime. Switch to axios (already a dependency, promoted to runtime) which provides timeout and HTTP-status errors natively, dropping the global fetch/AbortController usage.

Copilot

Pull request overview

Copilot reviewed 11 out of 11 changed files in this pull request and generated 1 comment.

Surface bitcoin.mempoolApiUrl in config-example.json so production deploys explicitly set the right network endpoint (mainnet default shown; testnet is https://mempool.space/testnet/api). Defaults for all keys live in src/lib/defaultConfig.js.

nicov-iov · 2026-06-25T17:42:59Z

Note: Closed and reopened PR to force reflect latest branch commit ede8004 (PR got desynced somehow)

Copilot

Pull request overview

Copilot reviewed 12 out of 12 changed files in this pull request and generated 2 comments.

The btc_merge_mining_stats table was added without bumping the schema version header / changelog. Record it as V1.2.6.

- Correct the BTC client header: it retries on network/timeout and 429/5xx (not other 4xx), and axios is used because a fetch global is absent before Node 18. - Stop the block walk early once remaining blocks can no longer reach the minimum coverage, so a provider outage doesn't drag the cron through the whole window.

Copilot

Pull request overview

Copilot reviewed 12 out of 12 changed files in this pull request and generated no new comments.

ivegabr · 2026-07-09T11:28:25Z

    "@rsksmart/rsk-contract-parser": "^2.2.0",
    "@rsksmart/rsk-js-cli": "^1.0.0",
    "@rsksmart/rsk-utils": "^1.1.0",
+    "axios": "^1.8.4",


instead of moving axios from devDependencies to dependencies, we can complete remove it, and use node fetch instead.

ivegabr · 2026-07-14T11:55:15Z

+import Logger from './Logger'
+
+const DEFAULTS = {
+  baseUrl: 'https://mempool.space/api',


This service is not reliable, almost every request fails due to time out. Consider using a reliable RPC service like Alchemy.

Also, the service runs once per day. If it fails due to time out, and after 3 retries, then, for that day, the stats will not be calculated. Consider adding an alternative service, or a fallback in case that for for that day all 3 retries have been exhausted, and there is no calculated hashrate.

see Bitcoin RCP documentation https://www.alchemy.com/docs/chains/bitcoin/bitcoin-api-endpoints/getblock

also, using devault JSON-RPC methods will allow us to switch providers easier.

https://developer.bitcoin.org/reference/rpc/getrawtransaction.html

By Getting the raw transaction of the conibase transaction we can find the RSK_TAG

ivegabr · 2026-07-14T11:57:45Z

@@ -0,0 +1,69 @@
+import axios from 'axios'


consider using node fetch instead of axios, and remove axios from the project.

Copilot AI review requested due to automatic review settings June 25, 2026 14:31

Copilot started reviewing on behalf of nicov-iov June 25, 2026 14:32 View session

Copilot AI reviewed Jun 25, 2026

View reviewed changes

Comment thread src/lib/btcMempoolClient.js Outdated

Comment thread src/lib/btcMempoolClient.js

Comment thread src/lib/btcMempoolClient.js

Comment thread src/lib/getMergeMiningStats.js

Comment thread test/rskMergeMiningTag.spec.js

Comment thread test/getMergeMiningStats.spec.js

nicov-iov added 2 commits June 25, 2026 13:51

fix(stats): validate sampling inputs and harden merge-mining fixtures

1d4355f

Reject non-positive sampleSize / out-of-range minCoverage so the ratio can never divide by zero and persist NaN. Use the correct OP_RETURN push opcode (0x29 = RSKBLOCK: + 32 bytes) in the test fixtures.

Copilot AI review requested due to automatic review settings June 25, 2026 16:59

Copilot started reviewing on behalf of nicov-iov June 25, 2026 17:00 View session

Copilot AI reviewed Jun 25, 2026

View reviewed changes

Comment thread prisma/rsk-explorer-database.sql

nicov-iov closed this Jun 25, 2026

nicov-iov reopened this Jun 25, 2026

Copilot AI review requested due to automatic review settings June 25, 2026 17:49

Copilot started reviewing on behalf of nicov-iov June 25, 2026 17:50 View session

Copilot AI reviewed Jun 25, 2026

View reviewed changes

Comment thread src/lib/btcMempoolClient.js Outdated

Comment thread src/lib/getMergeMiningStats.js

nicov-iov added 2 commits June 25, 2026 16:26

docs(db): bump schema version to V1.2.6 for merge-mining stats table

c519ab0

The btc_merge_mining_stats table was added without bumping the schema version header / changelog. Record it as V1.2.6.

Copilot AI review requested due to automatic review settings June 25, 2026 20:11

Copilot started reviewing on behalf of nicov-iov June 25, 2026 20:11 View session

Copilot AI reviewed Jun 25, 2026

View reviewed changes

nicov-iov requested a review from ivegabr July 7, 2026 14:05

ivegabr requested changes Jul 14, 2026

View reviewed changes

Uh oh!

Conversation

nicov-iov commented Jun 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What

How

Production database

Testing

Follow-up

Configuration (prod)

Uh oh!

github-actions Bot commented Jun 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Dependency Review

License Issues

package.json

OpenSSF Scorecard

Scanned Files

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

nicov-iov commented Jun 25, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

ivegabr Jul 9, 2026

Choose a reason for hiding this comment

Uh oh!

ivegabr Jul 14, 2026

Choose a reason for hiding this comment

Uh oh!

ivegabr Jul 14, 2026

Choose a reason for hiding this comment

Uh oh!

ivegabr Jul 14, 2026

Choose a reason for hiding this comment

Uh oh!

ivegabr Jul 14, 2026

Choose a reason for hiding this comment

Uh oh!

ivegabr Jul 14, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

nicov-iov commented Jun 25, 2026 •

edited

Loading

github-actions Bot commented Jun 25, 2026 •

edited

Loading