feat: support readable snapshots and independent Reader navigation

.changeset/bounded-physical-page-reads.md

@@ -0,0 +1,5 @@
+---
+'@ontos-ai/knowhere-sdk': minor
+---
+
+Add bounded physical-page reads and complete snapshot summaries for local and stored parsed snapshots. Select chunks before rich normalization and asset URL hydration, preserve physical pages and chunk identity, and expose a body-free document index with explicit embedded ownership for progressive readers. Add `knowledge.getDocumentSummary(reference)` to load the complete verified metadata index independently of rich chunk construction, table/image files, navigation bodies, and asset URL generation; remote-only references require a prepared or synchronized snapshot.

.changeset/initial-directory-read.md

@@ -0,0 +1,5 @@
+---
+'@ontos-ai/knowhere-sdk': minor
+---
+
+Add revision- and namespace-fenced initial directory reads without downloading document assets or triggering retrieval enhancement.

.changeset/readable-snapshot-image-origins.md

@@ -0,0 +1,7 @@
+---
+'@ontos-ai/knowhere-sdk': minor
+---
+
+Add durable readable snapshot preparation and authenticated, pinned image access before full image replication completes. Prefer certified local image replicas and keep archive checkpoints, completeness validation, and full-copy commits independent from the readable marker.
+
+Batch selected image URL capabilities through the immutable snapshot API, avoiding one API authorization request per page. Expose `getParsedDocumentObjectUrls` for authorized application resource readers without reading image bodies.

README.md

@@ -300,6 +300,14 @@ if (!documentId) {
 
 console.log(documentId);
 
+// Read all initially recognized chapters without waiting for Fine or copying assets.
+const directory = await client.documents.getStructure(documentId, {
+  namespace: 'support-center',
+});
+if (directory.directoryVersion === 1) {
+  console.log(directory.sections); // physical pageNums are 1-based
+}
+
 // Agentic mode (LLM navigation + answer synthesis)
 const response = await client.retrieval.query({
   namespace: 'support-center',
@@ -395,6 +403,16 @@ if (chunks.chunks[0]) {
 console.log(archived.status);
 ```
 
+Progressive parsing delivers `job.document.directory_ready` once logically per
+parsed revision, independently of baseline and final completion. Retries retain
+the same `event_id`; consumers should apply them idempotently and pass the
+event's `job_result_id` and `namespace` to `getStructure`. Version zero means
+the directory is pending. Version one includes every recognized descendant,
+including an empty directory when no hierarchy was found. A structure read
+does not trigger Fine or download page images. Raw webhook types are exported
+as `DocumentDirectoryReadyEvent`; SDK responses use camelCase and parse the
+directory timestamp into a `Date`.
+
 ### Local Knowledge Tools
 
 The SDK can also run exact inspection tools over parsed results. Local import
@@ -468,6 +486,121 @@ temporary disk space for ZIP integrity validation. See the
 [streaming snapshot guide](docs/streamed-parsed-document-sync.md) for the adapter
 contract, retry semantics, and memory validation harness.
 
+For immutable snapshots that support object delivery, establish readability before
+copying every image:
+
+```typescript
+const prepared = await knowledge.prepareReadableSnapshot({
+  jobId: 'job_123',
+  revisionKey: 'snapshot_baseline_123', // Always the snapshot ID, not the Job Result ID.
+  documentId: 'doc_123',
+  signal,
+});
+
+// Persist the selected revision in your durable workflow and publish your ready state.
+const page = await knowledge.readChunks({
+  documentId: prepared.documentId,
+  revisionKey: prepared.revisionKey,
+  page: 1,
+});
+
+// Resume image replication in a subsequent durable queue invocation.
+const copied = await knowledge.syncParsedDocument({
+  jobId: prepared.snapshot.jobId,
+  revisionKey: prepared.revisionKey,
+  signal,
+});
+// Re-enqueue the same revision while copied.completed is false.
+```
+
+`prepareReadableSnapshot` validates and stores every essential inventory object,
+including `manifest.json`, `chunks.json`, `doc_nav.json`, and `source.pdf`. It checks
+the pinned object inventory against document references and writes a separate
+`.knowhere-sdk/readable.json` descriptor. Interrupted core transfers reuse valid
+persisted objects on retry. Preparation does not write a full-copy commit, advance
+archive checkpoints, or change the current displayed snapshot. Readers must pass
+the selected `revisionKey` until the application publishes its own snapshot pointer.
+
+`readChunks` supplies local essential-object URLs after the readable barrier. Image
+URLs use fresh capabilities through the authenticated immutable snapshot while
+replication is incomplete; local image URLs are preferred after a full-copy commit
+certifies the same document, revision, and snapshot checksum. Conflicting commit
+identities are rejected without reading every image body to resolve metadata URLs.
+Applications can also call `getParsedDocumentObjectUrl({ documentId, revisionKey, path, signal })`
+or `getParsedDocumentObjectUrls({ documentId, revisionKey, paths, signal })` directly.
+The plural method returns a readonly record keyed by requested paths, with `null`
+for unavailable objects. It deduplicates paths and authorizes partial-copy image
+URLs in sequential batches of at most 200 through the authenticated snapshot API.
+`readChunks` batches only assets referenced by its selected chunks; URL resolution
+does not download image bodies. Legacy full copies use their existing local URLs.
+Applications can use `readParsedDocumentObject(...)` for bytes. Object reads prefer stored bytes and
+verify origin image length and SHA-256 before returning uncopied bytes. Origin
+reads do not copy images or certify replication. Capability URLs expire and must
+be refreshed rather than stored as permanent identities. The application must
+authorize its own document owner before exposing bytes or URLs to its readers.
+
+Use `knowledge.readChunks({ documentId, revisionKey, includeAssetUrls: false })`
+when the application resolves stored asset references through its own authorized
+resource reader. URL generation is enabled by default.
+
+Use `knowledge.getDocumentSummary(reference)` to build document navigation before
+loading any selected chunk body:
+
+```typescript
+const summary = await knowledge.getDocumentSummary({ documentId, revisionKey });
+console.log(summary.chunks); // Complete ordered metadata index, including unloaded chunks
+```
+
+The method reads only cached `manifest.json` and `chunks.json` after verifying a
+stored commit or readable snapshot identity. It does not construct rich chunks,
+read image/table files or navigation bodies, generate asset URLs, or scan remote
+chunks. A local cache ID (including a string reference) is also supported. A
+remote-only reference or unavailable stored revision rejects with `ValidationError`;
+prepare or synchronize the snapshot first. Job IDs alone do not locate a complete
+stored snapshot.
+
+For a complete local or stored snapshot, `readChunks` also returns
+`documentSummary`: the resolved revision, full chunk/type counts, trusted manifest
+page count (or `null`), sorted original physical page numbers, presentation flags,
+and an ordered `chunks` index containing IDs, positions, types, section/source
+paths, physical page numbers, and optional file paths. Index entries contain no
+body text or asset URLs. The summary covers the complete snapshot even when the
+returned chunk batch is filtered or paginated. Remote-only chunk windows omit it.
+Descriptors also expose optional `embeddedChunkIds` for explicit `embeds`
+connections, preserving chapter ownership for unloaded image/table entries.
+Unversioned local caches report `revisionKey: null`; a caller-supplied revision
+does not certify their contents. Stored snapshots report their verified revision.
+
+Use `sourcePageNumber` to read one original physical page without hydrating URLs
+for other pages:
+
+```typescript
+const page = await knowledge.readChunks({
+  documentId: prepared.documentId,
+  revisionKey: prepared.revisionKey,
+  sourcePageNumber: 137,
+  page: 1,
+  pageSize: 20,
+});
+```
+
+`sourcePageNumber` must be a positive safe integer. It filters before display
+pagination and URL resolution, retaining original chunk IDs and global positions.
+Multi-page chunks retain their content while their page-asset list is narrowed
+to the requested physical page. `totalChunks` and `totalPages` describe matching
+chunks and display batches; `documentSummary.pageNumbers` preserves the original
+physical page domain, including sparse pages. The stored JSON is still read and
+lightly scanned in full; only selected chunks receive rich normalization and URL
+hydration. Physical-page reads require a complete local or stored snapshot and
+throw `ValidationError` for remote-only documents; prepare or synchronize the
+pinned snapshot first. Ordinary remote reads keep their existing behavior.
+
+Only `syncParsedDocument(...).completed` and its existing full commit certify the
+complete archive copy and integrity checks. An origin or replication failure does
+not remove the readable descriptor. Historical snapshots without immutable object
+delivery return the API's unsupported/not-found error from preparation; callers
+can retain the existing full-archive synchronization path for those snapshots.
+
 If a search result only has a published `documentId`, read-oriented helpers can
 accept that remote identifier directly. `readChunks` supports display paging;
 remote fallback requests asset URLs from the Knowhere API, and storage hits use

scripts/publish-beta-release.mjs

@@ -31,6 +31,9 @@ if (!version.includes(`-${distTag}.`)) {
   throw new Error(`Beta publish expected ${version} to include prerelease tag ${distTag}`);
 }
 
+// Embed the assigned snapshot version rather than the workflow's earlier build version.
+releaseUtils.runCommand('pnpm', ['run', 'build:all']);
+
 // npm provenance is disabled while this repository is private.
 // Re-enable `--provenance` when the source repository becomes public.
 releaseUtils.publishWorkspacePackages({ tag: distTag });

src/__tests__/beta-release-contract.test.ts

@@ -1,5 +1,5 @@
-import { spawnSync } from 'node:child_process';
-import { mkdtemp, mkdir, readFile, writeFile, copyFile, rm } from 'node:fs/promises';
+import { spawnSync, type SpawnSyncReturns } from 'node:child_process';
+import { mkdtemp, mkdir, readFile, writeFile, copyFile, rm, symlink } from 'node:fs/promises';
 import { tmpdir } from 'node:os';
 import { delimiter, join } from 'node:path';
 import { expect, it } from 'vitest';
@@ -31,6 +31,8 @@ if (COMMAND === 'git' && args.join(' ') === 'rev-parse HEAD') {
     manifest.version = '2.6.1-beta.20990101000000';
     writeFileSync('package.json', JSON.stringify(manifest));
   }
+} else if (COMMAND === 'pnpm' && args[0] === 'run' && args[1] === 'build:all') {
+  // The separate package contract exercises actual build and pack bytes.
 } else if (COMMAND === 'pnpm' && args[0] === 'list') {
   console.log(JSON.stringify([{ path: process.cwd() }]));
 } else if (COMMAND === 'npm' && args[0] === 'view') {
@@ -79,3 +81,118 @@ if (COMMAND === 'git' && args.join(' ') === 'rev-parse HEAD') {
     await rm(directory, { recursive: true, force: true });
   }
 });
+
+it('publishes CommonJS and ESM bytes with the newly assigned beta package version', async (): Promise<void> => {
+  const directory: string = await mkdtemp(join(tmpdir(), 'knowhere-beta-package-contract-'));
+  const bin: string = join(directory, 'bin');
+  const scripts: string = join(directory, 'scripts');
+  const source: string = join(directory, 'src');
+  const temporary: string = join(directory, 'temporary');
+  const extracted: string = join(directory, 'extracted');
+  const capturePath: string = join(directory, 'published-package.json');
+  const expectedVersion: string = '2.8.0-beta.20990101000000';
+  try {
+    await Promise.all(
+      [bin, scripts, source, temporary, extracted].map(
+        (name: string): Promise<void> => mkdir(name),
+      ),
+    );
+    await Promise.all([
+      ...['publish-beta-release.mjs', 'release-utils.mjs'].map(
+        (name: string): Promise<void> =>
+          copyFile(join(process.cwd(), 'scripts', name), join(scripts, name)),
+      ),
+      copyFile(join(process.cwd(), 'package.json'), join(directory, 'package.json')),
+      copyFile(join(process.cwd(), 'tsconfig.json'), join(directory, 'tsconfig.json')),
+      copyFile(join(process.cwd(), 'src', 'version.ts'), join(source, 'version.ts')),
+      writeFile(join(source, 'index.ts'), "export { VERSION } from './version.js';\n"),
+      symlink(join(process.cwd(), 'node_modules'), join(directory, 'node_modules'), 'dir'),
+    ]);
+    // Match the workflow's build before Changesets assigns its snapshot version.
+    const initialBuild: SpawnSyncReturns<string> = spawnSync('pnpm', ['run', 'build'], {
+      cwd: directory,
+      encoding: 'utf8',
+    });
+    expect(initialBuild.status, initialBuild.stdout + initialBuild.stderr).toBe(0);
+
+    const commandBody: string = `
+import { execFileSync } from 'node:child_process';
+import { readFileSync, writeFileSync } from 'node:fs';
+import { createRequire } from 'node:module';
+import { join } from 'node:path';
+import { pathToFileURL } from 'node:url';
+const args = process.argv.slice(2);
+const originalEnvironment = { ...process.env, PATH: process.env.RELEASE_ORIGINAL_PATH };
+if (COMMAND === 'git' && args.join(' ') === 'rev-parse HEAD') {
+  console.log('bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb');
+} else if (COMMAND === 'pnpm' && args[0] === 'exec' && args[1] === 'changeset') {
+  if (args[2] === 'version') {
+    const manifest = JSON.parse(readFileSync('package.json', 'utf8'));
+    manifest.version = '${expectedVersion}';
+    writeFileSync('package.json', JSON.stringify(manifest));
+  }
+} else if (COMMAND === 'pnpm' && args[0] === 'run' && args[1] === 'build:all') {
+  execFileSync('pnpm', ['run', 'build'], { stdio: 'inherit', env: originalEnvironment });
+} else if (COMMAND === 'pnpm' && args[0] === 'list') {
+  console.log(JSON.stringify([{ path: process.cwd() }]));
+} else if (COMMAND === 'pnpm' && args[0] === 'pack') {
+  execFileSync('pnpm', args, { stdio: 'inherit', env: originalEnvironment });
+} else if (COMMAND === 'npm' && args[0] === 'view') {
+  process.exit(1);
+} else if (COMMAND === 'npm' && args[0] === 'publish') {
+  const extracted = process.env.RELEASE_EXTRACTED_PATH;
+  execFileSync('tar', ['-xzf', args[1], '-C', extracted]);
+  const packageDirectory = join(extracted, 'package');
+  const manifest = JSON.parse(readFileSync(join(packageDirectory, 'package.json'), 'utf8'));
+  const loadCommonJs = createRequire(import.meta.url);
+  const commonJs = loadCommonJs(join(packageDirectory, 'dist', 'index.js'));
+  const module = await import(pathToFileURL(join(packageDirectory, 'dist', 'index.mjs')).href);
+  writeFileSync(process.env.RELEASE_CAPTURE_PATH, JSON.stringify({
+    packageVersion: manifest.version,
+    commonJsVersion: commonJs.VERSION,
+    moduleVersion: module.VERSION,
+  }));
+} else if (COMMAND === 'gh' && args[0] === 'release' && args[1] === 'view') {
+  process.exit(0);
+} else {
+  throw new Error('Unexpected release boundary command: ' + COMMAND + ' ' + args.join(' '));
+}
+`;
+    await Promise.all(
+      ['git', 'pnpm', 'npm', 'gh'].map(
+        (name: string): Promise<void> =>
+          writeFile(
+            join(bin, name),
+            `#!/usr/bin/env node\nconst COMMAND = '${name}';\n${commandBody}`,
+            { mode: 0o755 },
+          ),
+      ),
+    );
+    const result: SpawnSyncReturns<string> = spawnSync(
+      process.execPath,
+      [join(scripts, 'publish-beta-release.mjs')],
+      {
+        cwd: directory,
+        encoding: 'utf8',
+        env: {
+          ...process.env,
+          PATH: `${bin}${delimiter}${process.env.PATH ?? ''}`,
+          TMPDIR: temporary,
+          BETA_DIST_TAG: 'beta',
+          RELEASE_ORIGINAL_PATH: process.env.PATH,
+          RELEASE_EXTRACTED_PATH: extracted,
+          RELEASE_CAPTURE_PATH: capturePath,
+        },
+      },
+    );
+    expect(result.status, result.stdout + result.stderr).toBe(0);
+    const captured: unknown = JSON.parse(await readFile(capturePath, 'utf8'));
+    expect(captured).toEqual({
+      packageVersion: expectedVersion,
+      commonJsVersion: expectedVersion,
+      moduleVersion: expectedVersion,
+    });
+  } finally {
+    await rm(directory, { recursive: true, force: true });
+  }
+}, 30_000);

src/__tests__/package-version-contract.test.ts

@@ -0,0 +1,17 @@
+import { readFile } from 'node:fs/promises';
+import { join } from 'node:path';
+import { expect, it } from 'vitest';
+import { VERSION } from '../index.js';
+import { NODE_SDK_DOCUMENT_METADATA_DEFAULTS } from '../lib/document-metadata.js';
+
+it('reports the package version in public exports and default document metadata', async (): Promise<void> => {
+  const manifest: unknown = JSON.parse(await readFile(join(process.cwd(), 'package.json'), 'utf8'));
+  if (typeof manifest !== 'object' || manifest === null || !('version' in manifest)) {
+    throw new Error('Expected the SDK package manifest to declare its version');
+  }
+
+  expect({
+    exportedVersion: VERSION,
+    clientVersion: NODE_SDK_DOCUMENT_METADATA_DEFAULTS.clientVersion,
+  }).toEqual({ exportedVersion: manifest.version, clientVersion: manifest.version });
+});

src/index.ts

@@ -20,8 +20,17 @@ export type {
   JobResult,
   JobStatus,
   JobError,
+  JobSnapshot,
+  JobSnapshotObject,
+  JobSnapshotDelivery,
+  JobSnapshotObjectAccess,
+  JobSnapshotObjectAccessBatch,
   // Document types
   Document,
+  DocumentDirectoryParams,
+  DocumentDirectoryReadyEvent,
+  DocumentDirectoryResponse,
+  DocumentDirectorySection,
   DocumentChunk,
   DocumentChunkGetParams,
   DocumentChunkListParams,
@@ -69,6 +78,7 @@ export type {
   KnowhereAssetStorageWriteResult,
   ParsedDocumentCommit,
   ParsedDocumentCommitSource,
+  ParsedDocumentReadableSnapshot,
   ParsedDocumentObject,
   ParsedDocumentObjectHead,
   ParsedDocumentObjectParams,
@@ -116,6 +126,8 @@ export type {
   KnowledgeReadParams,
   KnowledgeReadChunk,
   KnowledgeReadResponse,
+  KnowledgeChunkDescriptor,
+  KnowledgeDocumentSummary,
   KnowledgeGrepParams,
   KnowledgeGrepMatch,
   KnowledgeGrepResponse,
@@ -126,6 +138,10 @@ export type {
   KnowledgeStartupRecoveryResponse,
   KnowledgeSyncParsedDocumentParams,
   KnowledgeSyncParsedDocumentResponse,
+  KnowledgePrepareReadableSnapshotParams,
+  KnowledgePrepareReadableSnapshotResponse,
+  KnowledgeReadParsedDocumentObjectParams,
+  KnowledgeReadParsedDocumentObjectUrlsParams,
 } from './types/index.js';
 
 export { storeParseResultAssets } from './storage/asset-storage.js';