docs: Add reliable transaction pagination guidance and example

README.md

@@ -295,26 +295,69 @@ To query based on a historical timestamp, you will need to convert the timstamp
 
 ### Pagination
 
-The Graph places a limit on the number of items which can returned by a single query (currently 100). To fetch a larger number of items, the `first` and `skip` parameters can be used to create paginated queries.
+The Graph returns 100 entities per collection by default; this is not a hard maximum of 100 per query. Set `first` explicitly and use a page size supported by the endpoint. See [The Graph's querying best practices](https://thegraph.com/docs/en/subgraphs/querying/best-practices/).
 
-For example, if you wanted to fetch the first 200 transactions for an Aave market, you can't query 200 items at once, but you can achieve the same thing by concatenating the output of these queries:
+`first` and `skip` work for small offset-based queries, but large `skip` values generally perform poorly. For larger exports, use an ID cursor with `orderBy: id`, `orderDirection: asc`, and `where: { id_gt: $lastID }`, as described in [The Graph's GraphQL API guide](https://thegraph.com/docs/en/subgraphs/querying/graphql-api/#pagination-example).
 
-```
-{
-  userTransactions(orderBy: timestamp, orderDirection: asc, first: 100, skip: 0){
+#### Exporting a User's Transaction History
+
+Use a V2 or V3 protocol deployment exposing `UserTransaction`, not a governance subgraph. Choose a finalized block hash on that deployment's network, and keep it fixed for every page. The endpoint must have indexed and retained the requested block; an unavailable historical block is an error, not an empty result.
+
+```graphql
+query TransactionPage($first: Int!, $lastID: String!, $user: String!, $blockHash: Bytes!) {
+  userTransactions(
+    first: $first
+    orderBy: id
+    orderDirection: asc
+    where: { user: $user, id_gt: $lastID }
+    block: { hash: $blockHash }
+  ) {
+    id
     timestamp
+    txHash
+    action
+  }
+  _meta(block: { hash: $blockHash }) {
+    deployment
+    hasIndexingErrors
+    block { number hash }
   }
 }
 ```
 
+Start with `lastID: ""`, a lowercase wallet address for `user`, and an explicit page size such as `first: 100`. After each page, pass its final `id` as `lastID`. Continue until an empty page is returned, even if an earlier page contains fewer rows than requested.
+
+IDs are ordered lexicographically, not chronologically. If chronological output is needed, sort the collected rows by `timestamp`, using `id` to break ties. A timestamp alone is not a safe cursor because multiple events can share the same timestamp.
+
+#### Read-only Node.js Example
+
+[`examples/paginate-user-transactions.mjs`](examples/paginate-user-transactions.mjs) implements this loop using native `fetch` on Node.js 20.18.1 or later, with no dependency installation required. Replace all placeholders below with your query endpoint, wallet address, and finalized block hash. Obtain the query URL and any required API key from The Graph Explorer; do not include an authenticated URL in public reports or logs.
+
+```bash
+export SUBGRAPH_URL='https://YOUR_QUERY_ENDPOINT'
+export USER_ADDRESS='0xYOUR_WALLET_ADDRESS'
+export BLOCK_HASH='0xYOUR_FINALIZED_BLOCK_HASH'
+node examples/paginate-user-transactions.mjs
 ```
-{
-  userTransactions(orderBy: timestamp, orderDirection: asc, first: 100, skip: 100){
-    timestamp
-  }
-}
+
+The command prints the transaction count and snapshot metadata after the loop finishes. `PAGE_SIZE` optionally overrides the default of 100. The exported `paginateUserTransactions` async generator also yields each page for applications that need the transaction rows.
+
+Run the example's local tests with:
+
+```bash
+node --test examples/paginate-user-transactions.test.mjs
 ```
 
+#### Checking an Export
+
+- Check HTTP status and GraphQL errors before reading a page; do not accept partial data alongside errors.
+- Check `_meta.deployment`, `_meta.block.number`, and `_meta.block.hash` on every page, including the empty final page. Restart if the deployment or snapshot changes.
+- Require `_meta.hasIndexingErrors` to be `false`. Missing metadata or an indexing error is not evidence of a complete export.
+- Reject duplicate, unordered, or non-advancing IDs rather than silently deduplicating inconsistent responses. The example performs these checks before yielding each page.
+- Treat any failed run as incomplete, even if earlier pages were collected successfully.
+
+Exhausting this cursor covers the matching entities returned by that indexed deployment at the selected block. It does not independently prove that every on-chain event was indexed correctly. For inconsistent results, record the exact query, variables, deployment ID, block number/hash, page counts, and error responses, with credentials removed. Compare repeated exports against the same deployment and snapshot before attributing missing data to pagination, indexing, or backend behaviour.
+
 </details>
 
 ## Development

examples/paginate-user-transactions.mjs

@@ -0,0 +1,116 @@
+import { pathToFileURL } from 'node:url';
+
+export const TRANSACTIONS_QUERY = `
+  query TransactionPage($first: Int!, $lastID: String!, $user: String!, $blockHash: Bytes!) {
+    userTransactions(
+      first: $first
+      orderBy: id
+      orderDirection: asc
+      where: { user: $user, id_gt: $lastID }
+      block: { hash: $blockHash }
+    ) {
+      id
+      timestamp
+      txHash
+      action
+    }
+    _meta(block: { hash: $blockHash }) {
+      deployment
+      hasIndexingErrors
+      block { number hash }
+    }
+  }
+`;
+
+export async function* paginateUserTransactions(
+  { url, user, blockHash, first = 100 },
+  request = globalThis.fetch
+) {
+  if (!url || !['https:', 'http:'].includes(new URL(url).protocol)) {
+    throw new Error('SUBGRAPH_URL must be an HTTP(S) GraphQL endpoint');
+  }
+  if (!/^0x[0-9a-f]{40}$/i.test(user || '')) {
+    throw new Error('USER_ADDRESS must be a 20-byte Ethereum address');
+  }
+  if (!/^0x[0-9a-f]{64}$/i.test(blockHash || '')) {
+    throw new Error('BLOCK_HASH must be a 32-byte block hash');
+  }
+  if (!Number.isInteger(first) || first < 1) {
+    throw new Error('PAGE_SIZE must be a positive integer supported by the endpoint');
+  }
+
+  user = user.toLowerCase();
+  blockHash = blockHash.toLowerCase();
+  let lastID = '';
+  let deployment;
+  let blockNumber;
+
+  while (true) {
+    const response = await request(url, {
+      method: 'POST',
+      headers: { 'content-type': 'application/json' },
+      body: JSON.stringify({
+        query: TRANSACTIONS_QUERY,
+        variables: { first, lastID, user, blockHash },
+      }),
+      signal: AbortSignal.timeout(30_000),
+    });
+    if (!response.ok) throw new Error(`GraphQL HTTP error ${response.status}`);
+    const result = await response.json();
+    if (result.errors?.length) throw new Error('GraphQL query returned errors');
+
+    const meta = result.data?._meta;
+    if (
+      typeof meta?.deployment !== 'string' ||
+      !meta.deployment ||
+      meta.hasIndexingErrors !== false ||
+      meta.block?.hash?.toLowerCase() !== blockHash ||
+      !Number.isInteger(meta.block?.number) ||
+      meta.block.number < 0
+    ) {
+      throw new Error('Missing or unsafe snapshot metadata; do not treat this export as complete');
+    }
+    if (
+      deployment !== undefined &&
+      (meta.deployment !== deployment || meta.block.number !== blockNumber)
+    ) {
+      throw new Error('Snapshot changed between pages; restart the export');
+    }
+    deployment = meta.deployment;
+    blockNumber = meta.block.number;
+
+    const transactions = result.data?.userTransactions;
+    if (!Array.isArray(transactions)) throw new Error('Missing transaction page');
+    let nextID = lastID;
+    for (const transaction of transactions) {
+      if (typeof transaction.id !== 'string' || transaction.id <= nextID) {
+        throw new Error('Duplicate or unordered IDs; cursor did not advance');
+      }
+      nextID = transaction.id;
+    }
+
+    yield { transactions, deployment, block: meta.block };
+    if (transactions.length === 0) return;
+    lastID = nextID;
+  }
+}
+
+if (process.argv[1] && import.meta.url === pathToFileURL(process.argv[1]).href) {
+  try {
+    let count = 0;
+    let snapshot;
+    for await (const page of paginateUserTransactions({
+      url: process.env.SUBGRAPH_URL,
+      user: process.env.USER_ADDRESS,
+      blockHash: process.env.BLOCK_HASH,
+      first: Number(process.env.PAGE_SIZE || 100),
+    })) {
+      count += page.transactions.length;
+      snapshot = { deployment: page.deployment, block: page.block };
+    }
+    console.log(JSON.stringify({ count, ...snapshot }, null, 2));
+  } catch (error) {
+    console.error(error.message);
+    process.exitCode = 1;
+  }
+}

examples/paginate-user-transactions.test.mjs

@@ -0,0 +1,196 @@
+import assert from 'node:assert/strict';
+import test from 'node:test';
+import { spawnSync } from 'node:child_process';
+import { fileURLToPath } from 'node:url';
+import { readFileSync } from 'node:fs';
+import { paginateUserTransactions, TRANSACTIONS_QUERY } from './paginate-user-transactions.mjs';
+
+const blockHash = `0x${'a'.repeat(64)}`;
+const user = `0x${'b'.repeat(40)}`;
+const options = { url: 'https://example.invalid/graphql', user, blockHash, first: 2 };
+const row = (id) => ({ id, timestamp: 1, txHash: '0x00', action: 'Supply' });
+const metadata = () => ({
+  deployment: 'test-deployment',
+  hasIndexingErrors: false,
+  block: { number: 123, hash: blockHash },
+});
+const payload = (ids, meta = metadata()) => ({
+  data: { userTransactions: ids.map(row), _meta: meta },
+});
+
+test('README query matches the tested example', () => {
+  const readme = readFileSync(new URL('../README.md', import.meta.url), 'utf8');
+  const queries = [...readme.matchAll(/```graphql\s*([\s\S]*?)```/g)];
+  assert.ok(queries.some((match) =>
+    match[1].replace(/\s+/g, '') === TRANSACTIONS_QUERY.replace(/\s+/g, '')));
+});
+
+test('selected transaction fields and user relation exist in both protocol schemas', () => {
+  for (const version of ['v2', 'v3']) {
+    const schema = readFileSync(new URL(`../schemas/${version}.schema.graphql`, import.meta.url),
+      'utf8');
+    const transaction = schema.match(/interface UserTransaction\s*\{([^}]+)\}/)[1];
+    for (const field of ['id: ID!', 'timestamp: Int!', 'txHash: Bytes!', 'action: Action!',
+      'user: User!']) assert.ok(transaction.includes(field), `${version} missing ${field}`);
+  }
+});
+
+function mockRequest(results) {
+  const calls = [];
+  const request = async (url, init) => {
+    calls.push({ url, init, body: JSON.parse(init.body) });
+    const result = results[calls.length - 1];
+    assert.ok(result, 'unexpected request: paginator must terminate');
+    return { ok: true, status: 200, json: async () => result };
+  };
+  return { calls, request };
+}
+
+async function collect(config, request) {
+  const pages = [];
+  for await (const page of paginateUserTransactions(config, request)) pages.push(page);
+  return pages;
+}
+
+test('cursor export preserves tied timestamps and pins every request to one snapshot', async () => {
+  const mock = mockRequest([payload(['a', 'b']), payload(['c', 'd']), payload([])]);
+  const pages = await collect(options, mock.request);
+  assert.deepEqual(pages.flatMap((page) => page.transactions.map((entry) => entry.id)),
+    ['a', 'b', 'c', 'd']);
+  assert.deepEqual(mock.calls.map((call) => call.body.variables.lastID), ['', 'b', 'd']);
+  for (const call of mock.calls) {
+    assert.equal(call.init.method, 'POST');
+    assert.equal(call.body.query, TRANSACTIONS_QUERY);
+    assert.equal(call.body.variables.blockHash, blockHash);
+    assert.equal(call.body.variables.user, user);
+    assert.equal(call.body.variables.first, 2);
+  }
+});
+
+test('a short page does not end an export; continue until an empty page', async () => {
+  const mock = mockRequest([payload(['a']), payload(['b']), payload([])]);
+  const pages = await collect({ ...options, first: 100 }, mock.request);
+  assert.equal(pages.flatMap((page) => page.transactions).length, 2);
+  assert.equal(mock.calls.length, 3);
+});
+
+test('an empty initial page retains snapshot metadata', async () => {
+  const mock = mockRequest([payload([])]);
+  const pages = await collect(options, mock.request);
+  assert.equal(pages.length, 1);
+  assert.deepEqual(pages[0].block, metadata().block);
+  assert.equal(pages[0].transactions.length, 0);
+});
+
+test('normalise checksum-address and block-hash case without changing opaque IDs', async () => {
+  const mock = mockRequest([payload(['a']), payload([])]);
+  await collect({ ...options, user: user.toUpperCase().replace('0X', '0x'),
+    blockHash: blockHash.toUpperCase().replace('0X', '0x') }, mock.request);
+  assert.equal(mock.calls[0].body.variables.user, user);
+  assert.equal(mock.calls[0].body.variables.blockHash, blockHash);
+});
+
+test('reject invalid configuration before any request', async () => {
+  for (const invalid of [{ url: undefined }, { url: 'file:///tmp/data' }, { user: 'invalid' },
+    { blockHash: '0x01' }, { first: 0 }, { first: 1.5 }, { first: NaN }]) {
+    await assert.rejects(collect({ ...options, ...invalid }, () => {
+      assert.fail('invalid configuration must not contact an endpoint');
+    }));
+  }
+});
+
+test('HTTP errors and transport failures are not empty pages', async () => {
+  await assert.rejects(collect(options, async () => ({ ok: false, status: 429 })),
+    /HTTP error 429/);
+  await assert.rejects(collect(options, async () => { throw new Error('transport failed'); }),
+    /transport failed/);
+});
+
+test('GraphQL errors reject partial data', async () => {
+  const mock = mockRequest([{ ...payload(['a']), errors: [{ message: 'block unavailable' }] }]);
+  await assert.rejects(collect(options, mock.request), /returned errors/);
+});
+
+test('reject indexing errors, missing metadata and a different block hash', async () => {
+  for (const meta of [null, { ...metadata(), hasIndexingErrors: true },
+    { ...metadata(), deployment: '' }, { ...metadata(), hasIndexingErrors: undefined },
+    { ...metadata(), block: { number: 123, hash: null } },
+    { ...metadata(), block: { number: 123, hash: `0x${'c'.repeat(64)}` } }]) {
+    const mock = mockRequest([payload(['a'], meta)]);
+    await assert.rejects(collect(options, mock.request), /snapshot metadata/);
+  }
+});
+
+test('reject a deployment or block-number change, including on the empty final page', async () => {
+  for (const meta of [{ ...metadata(), deployment: 'different-deployment' },
+    { ...metadata(), block: { number: 124, hash: blockHash } }]) {
+    const mock = mockRequest([payload(['a']), payload([], meta)]);
+    await assert.rejects(collect(options, mock.request), /Snapshot changed/);
+  }
+});
+
+test('reject missing transaction data rather than claiming no results', async () => {
+  for (const transactions of [null, undefined, {}]) {
+    const mock = mockRequest([{ data: { _meta: metadata(), userTransactions: transactions } }]);
+    await assert.rejects(collect(options, mock.request), /Missing transaction page/);
+  }
+});
+
+test('reject duplicate, unordered and non-advancing cursors', async () => {
+  for (const results of [[payload(['a', 'a'])], [payload(['b', 'a'])],
+    [payload(['a']), payload(['a'])], [payload(['b']), payload(['a'])]]) {
+    const mock = mockRequest(results);
+    await assert.rejects(collect(options, mock.request), /Duplicate or unordered/);
+  }
+});
+
+test('reject invalid transaction IDs', async () => {
+  for (const id of [null, undefined, 1, '']) {
+    const mock = mockRequest([payload([id])]);
+    await assert.rejects(collect(options, mock.request), /Duplicate or unordered/);
+  }
+});
+
+test('CLI validates configuration and exits non-zero without printing a success summary', () => {
+  const result = spawnSync(process.execPath,
+    [fileURLToPath(new URL('./paginate-user-transactions.mjs', import.meta.url))],
+    { encoding: 'utf8', env: {
+      ...process.env, SUBGRAPH_URL: '', USER_ADDRESS: '', BLOCK_HASH: '',
+    } });
+  assert.equal(result.status, 1);
+  assert.equal(result.stdout, '');
+  assert.match(result.stderr, /SUBGRAPH_URL/);
+});
+
+test('CLI prints a summary only after reaching the empty final page', () => {
+  const results = [payload(['a', 'b']), payload(['c']), payload([])];
+  const bootstrap = `const results = ${JSON.stringify(results)};
+    globalThis.fetch = async () => {
+      const result = results.shift();
+      if (!result) throw new Error('unexpected request');
+      return { ok: true, json: async () => result };
+    };`;
+  const result = spawnSync(process.execPath,
+    ['--import', `data:text/javascript,${encodeURIComponent(bootstrap)}`,
+      fileURLToPath(new URL('./paginate-user-transactions.mjs', import.meta.url))],
+    { encoding: 'utf8', env: { ...process.env, SUBGRAPH_URL: options.url,
+      USER_ADDRESS: user, BLOCK_HASH: blockHash, PAGE_SIZE: '2' } });
+  assert.equal(result.status, 0, result.stderr);
+  assert.deepEqual(JSON.parse(result.stdout), {
+    count: 3, deployment: 'test-deployment', block: metadata().block,
+  });
+});
+
+test('CLI rejects a later failed page without printing partial success', () => {
+  const results = [payload(['a']), { errors: [{ message: 'index unavailable' }] }];
+  const bootstrap = `const results = ${JSON.stringify(results)};
+    globalThis.fetch = async () => ({ ok: true, json: async () => results.shift() });`;
+  const result = spawnSync(process.execPath,
+    ['--import', `data:text/javascript,${encodeURIComponent(bootstrap)}`,
+      fileURLToPath(new URL('./paginate-user-transactions.mjs', import.meta.url))],
+    { encoding: 'utf8', env: { ...process.env, SUBGRAPH_URL: options.url,
+      USER_ADDRESS: user, BLOCK_HASH: blockHash, PAGE_SIZE: '2' } });
+  assert.equal(result.status, 1);
+  assert.equal(result.stdout, '');
+  assert.match(result.stderr, /returned errors/);
+});