diff --git a/docs/audit-remediation-status.md b/docs/audit-remediation-status.md deleted file mode 100644 index 107170d..0000000 --- a/docs/audit-remediation-status.md +++ /dev/null @@ -1,49 +0,0 @@ -# Audit Remediation Status - -## All Phases Complete - -### Completed -- **Phase 1A**: `npm audit fix` — all 7 vulnerabilities resolved (0 remaining) -- **nvm hook**: Added `.claude/hooks/init-nvm.sh` + `.claude/settings.json` so future sessions auto-source nvm -- **Phase 1B**: ESLint + Prettier configs verified working - - `.prettierignore` added to exclude generated content, lock files, test output - - All source files formatted; `npm run lint` and `npm run prettier` pass clean -- **Phase 2**: Removed ~310 lines of dead code (`syncEntry`, `processDirectoryRecursively`) - - Cleaned up unused imports (`INVALID_SERVICE_RESPONSE`, `INVALID_URL_ERROR`, `RenderedContent`) - - `github.content.ts` reduced from 1406 → 1096 lines -- **Phase 3**: Replaced `octokit: any` → `Octokit` in 8 functions across 4 files - - Fixed type narrowing bug in `downloadAsset` (array guard for `repos.getContent` union type) -- **Phase 4**: Defined `ExtendedLoaderContext` type, eliminated `context: any` and logger casts - - `ExtendedLoaderContext = Omit & { logger: Logger }` - - `CollectionEntryOptions.context` now uses `ExtendedLoaderContext` - - Removed `as unknown as Logger` and `as any` logger casts -- **Phase 5A**: `error: any` → `error: unknown` in 8 catch blocks across 4 files - - `error instanceof Error ? error.message : String(error)` for message access - - Proper type guards for `.status` checks (Octokit errors) -- **Phase 5B**: Replaced bare `console.*` with Logger in cleanup and dryrun modules - - Threaded logger through `getExpectedFiles`, `getExistingFiles`, `loadImportState`, `saveImportState` - - `eslint-disable` for legitimate startup messages in `github.auth.ts` (no logger available) - - Lint warnings: 7 → 0 -- **Phase 5C**: Cleaned up remaining `any` types in production code - - `treeData.tree.filter` item type annotation - - `transformsToApply: TransformFunction[]` - - `displayDryRunResults` logger parameter typed - - `LinkMapping.replacement` callback context typed as `LinkTransformContext` - - Auto-generated link mapping callbacks properly typed - -### Remaining `any` (test files only) -- `github.content.spec.ts`: `ctx as any` for mock Astro context (8 instances) -- `test-helpers.ts`: `as any` for mock fixtures (6 instances) -- `github.types.ts`: `frontmatter?: Record` — matches Astro's own type - -## Verification -- `npx tsc --noEmit` — clean compile -- `npm test` — 20/20 tests pass -- `npm run lint` — 0 warnings -- `npm run prettier` — all files formatted - -## Key Files -- Main source: `packages/astro-github-loader/src/github.content.ts` (1096 lines) -- Types: `packages/astro-github-loader/src/github.types.ts` (includes `ExtendedLoaderContext`) -- Logger: `packages/astro-github-loader/src/github.logger.ts` -- Loader orchestration: `packages/astro-github-loader/src/github.loader.ts` diff --git a/docs/loader-redesign-prompt.md b/docs/loader-redesign-prompt.md deleted file mode 100644 index 5f92e7c..0000000 --- a/docs/loader-redesign-prompt.md +++ /dev/null @@ -1,133 +0,0 @@ -# Loader Redesign — Session Prompt - -Use this document as context for a Claude Code session in the `astro-github-loader` repo. It captures design decisions made in the devportal repo that affect the loader. - -## Background - -The Algorand Developer Portal (`devportal`) uses `@larkiny/astro-github-loader` to import documentation from GitHub repositories into Astro content collections. We are redesigning the devportal's library configuration system. Some of those changes require corresponding updates to the loader. - -The devportal design doc is at: `/Users/larkinyoung/dev/af/devportal/docs/planning/2026-02-12-library-config-redesign.md` - -## What the devportal is changing - -1. **Import configs now use a `variants` wrapper.** Each library has a single config file with a `metadata` object and a `variants` array. Each variant represents a language-specific import (e.g., TypeScript, Python) from a different GitHub repo. The loader receives individual variants to process — same as today's `ImportOptions`, but with added `language` and `versions` fields. - -2. **Versions are per-variant, declared in the import config.** Each variant has a `versions` array listing the versions to display (e.g., `[{ slug: 'latest', label: 'Latest' }, { slug: 'v8.0.0', label: 'v8.0.0' }]`). Versions are manually curated — no auto-discovery, no manifest files. The version folders already exist in the source repo's content (typically on a `docs-dist` branch). - -3. **Assets are co-located with content.** Instead of a separate `assetsPath` directory, assets (images, etc.) should be placed in an `assets/` folder within the content destination path for each version. This means `assetsPath` and `assetsBaseUrl` can be derived from the variant's content `basePath` rather than explicitly configured. - -## Changes needed in the loader - -### 1. ImportOptions type updates - -**What:** Add `language` and `versions` fields directly to the existing `ImportOptions` type. No backward-compatibility shim needed — the devportal is the only consumer of this loader. - -```typescript -interface VersionConfig { - slug: string; // URL segment: "latest", "v8.0.0" - label: string; // Display name: "Latest", "v8.0.0" -} - -// Added to ImportOptions: -language?: string; // "TypeScript", "Python", "Go", etc. -versions?: VersionConfig[]; // Versions to display in the devportal's version picker -``` - -The `language` field is useful for logging. The `versions` field is informational — it tells the loader which version folders exist in the source content, which can be used for validation or path mapping. The core import logic doesn't change based on these fields. - -### 2. Co-located assets - -**What:** When the loader downloads assets (images referenced in markdown), place them in an `assets/` subdirectory relative to the content's destination path, rather than in a separate `assetsPath` directory. - -**Current behavior:** -- Content goes to: `src/content/docs/docs/algokit-utils/typescript/` -- Assets go to: `src/assets/algokit-utils-ts/` (separate directory) -- Markdown references: absolute path to assets directory - -**New behavior:** -- Content goes to: `src/content/docs/docs/algokit-utils/typescript/v8.0.0/` -- Assets go to: `src/content/docs/docs/algokit-utils/typescript/v8.0.0/assets/` -- Markdown references: relative path (`./assets/image.png`) - -**Benefits:** -- `assetsPath` and `assetsBaseUrl` can be removed from the config (derived from `basePath`) -- Version cleanup is atomic (delete version folder, assets go with it) -- Relative references are simpler - -**Backward compatibility:** If `assetsPath` is explicitly provided, use it (existing behavior). If omitted, default to `{basePath}/assets/`. - -### 3. Version-aware content import - -**What:** The source repos publish documentation to a `docs-dist` branch (or equivalent) with version folders already in place. The loader imports from a single ref — it does NOT need to check out different branches/tags per version. - -**Source content structure (on the docs-dist branch):** -``` -latest/ ← always present - getting-started/ - guides/ - api/ -v8.0.0/ ← pinned release version (optional) - getting-started/ - guides/ - api/ -``` - -**Destination path structure:** -``` -src/content/docs/docs/algokit-utils/typescript/ - latest/ - getting-started/ - guides/ - assets/ - v8.0.0/ - getting-started/ - guides/ - assets/ -``` - -**Key point:** The loader does NOT iterate over multiple refs. It imports from one ref (the docs-dist branch), and the version folder structure carries through from source to destination. The `versions` field in the config tells the devportal which versions exist for the UI (version picker), but the loader's job is simply to import the content as it finds it. - -**For external repos** that don't use the docs-dist convention: the `ref` field on the import config points to whatever branch contains the docs, and typically only a single version (`latest`) is configured. The loader doesn't need special handling — it just imports from the configured ref as it does today. - -## What does NOT change in the loader - -- **Core import pipeline** — file discovery, glob matching, content transforms, path mappings all stay the same -- **Link transformation** — same mechanism, just applied per-version -- **Authentication** — same GitHub App / PAT approach -- **Change detection** — same ref-aware caching - -## Suggested approach - -1. **Start by reading the current loader codebase** to understand the import pipeline, especially how `ref`, `basePath`, and asset handling work. -2. **Update `ImportOptions` type** — add `language` and `versions` fields. Simplest change, do first. -3. **Design co-located assets** — default `assetsPath` to `{basePath}/assets/` when not explicitly provided. - -## Testing improvements - -The current test setup needs significant improvement. Before making any functional changes, review the existing test infrastructure and bring it up to best practices. This is a priority — all loader changes should ship with solid test coverage. - -**Audit the current setup:** -- Review the test framework, runner config, and existing test files -- Identify what's covered and what's not (especially core pipeline functions, transforms, asset handling, and path mapping) -- Check for proper mocking of GitHub API calls (no real network requests in tests) -- Assess whether tests are unit-level, integration-level, or a mix - -**Apply these practices:** -- **Mock external dependencies.** All GitHub API interactions should be mocked. Tests must run offline and fast. -- **Use fixtures for file content.** Create a `test/fixtures/` directory with sample markdown files, images, and directory structures that represent real import scenarios. -- **Test transforms in isolation.** Each content transform (frontmatter injection, link rewriting, path mapping, etc.) should have its own focused tests with clear input/output assertions. -- **Test the asset pipeline.** Verify that assets are downloaded, placed in the correct directory, and that markdown references are rewritten correctly — especially for the new co-located assets default. -- **Test config validation.** Verify that malformed or incomplete `ImportOptions` are handled gracefully (missing required fields, invalid patterns, etc.). -- **Add coverage reporting.** Configure the test runner to output coverage metrics. Aim for meaningful coverage of core logic — don't chase 100%, but ensure critical paths (file discovery, glob matching, content writing, path resolution) are well-covered. -- **Make tests easy to run.** `npm test` (or equivalent) should run the full suite with no setup required. Add a `test:watch` script for development. - -**For each loader change in this session**, write tests alongside the implementation: -1. `ImportOptions` type updates — test that the new fields are accepted and passed through correctly -2. Co-located assets — test the default `{basePath}/assets/` behavior when `assetsPath` is omitted, and that explicit `assetsPath` still works - -## Reference files in devportal - -- Design doc: `/Users/larkinyoung/dev/af/devportal/docs/planning/2026-02-12-library-config-redesign.md` -- Current import configs: `/Users/larkinyoung/dev/af/devportal/imports/configs/` -- Current types: `/Users/larkinyoung/dev/af/devportal/imports/transforms/types.ts` -- Content config: `/Users/larkinyoung/dev/af/devportal/src/content/config.ts` diff --git a/packages/astro-github-loader/PROCESSING_FLOW.md b/packages/astro-github-loader/PROCESSING_FLOW.md deleted file mode 100644 index 26c2aac..0000000 --- a/packages/astro-github-loader/PROCESSING_FLOW.md +++ /dev/null @@ -1,309 +0,0 @@ -# Astro GitHub Loader - Content Processing Pipeline - -This document provides a comprehensive overview of how content flows through the Astro GitHub Loader, from initial file discovery to final output in your Astro content collection. - -## Overview - -The Astro GitHub Loader processes content in four main phases: - -1. **File Discovery and Collection** - Scan GitHub repository and identify files to import -2. **Individual File Processing** - Transform each file's content and determine target paths -3. **Global Link Transformation** - Resolve and transform links between imported files -4. **File Storage** - Write processed files to Astro content store - -## Phase 1: File Discovery and Collection - -### Step 1: Directory Scanning -- **Function**: `collectFilesRecursively()` -- **Input**: Include patterns from config -- **Process**: Recursively scans GitHub repo directories -- **Configuration Used**: `includes[].pattern` (glob patterns) -- **Output**: List of files matching include patterns - -**Example**: -```typescript -// Configuration -includes: [{ - pattern: "docs/markdown/autoapi/algokit_utils/**/*", - basePath: "src/content/docs/reference/algokit-utils-py/api" -}] - -// Output -[ - "docs/markdown/autoapi/algokit_utils/index.md", - "docs/markdown/autoapi/algokit_utils/applications/abi/index.md", - "docs/markdown/autoapi/algokit_utils/applications/app_client/AppClient.md", - // ... -] -``` - -### Step 2: File Content Fetching -- **Function**: `collectFileData()` -- **Process**: Downloads raw content from GitHub -- **Output**: Raw file content as string - -## Phase 2: Individual File Processing - -### Step 3: Asset Processing -- **Function**: `processAssets()` -- **Configuration Used**: `assetsPath`, `assetsBaseUrl`, `assetPatterns` -- **Process**: Downloads and transforms asset references (images, etc.) - -**Example**: -```markdown -// Before asset processing -![Diagram](./diagram.png) - -// After asset processing -![Diagram](@assets/imports/algokit-utils-py/diagram.png) -``` - -### Step 4: Path Mapping Application -- **Function**: `generatePath()` -- **Configuration Used**: `includes[].pathMappings` -- **Process**: Determines final target path for each file - -**Examples**: -```typescript -// Configuration -pathMappings: { - "docs/markdown/autoapi/algokit_utils/": "", - "docs/markdown/index.md": "overview.md" -} - -// Transformations -"docs/markdown/autoapi/algokit_utils/index.md" -→ "src/content/docs/reference/algokit-utils-py/api/index.md" - -"docs/markdown/index.md" -→ "src/content/docs/algokit/utils/python/overview.md" -``` - -### Step 5: Content Transformations -- **Configuration Used**: - - `transforms[]` (global transforms) - - `includes[].transforms[]` (pattern-specific transforms) -- **Process**: Apply content transformation functions - -**Examples**: -```typescript -// Transform functions -convertH1ToTitle: "# My Title" → frontmatter: {title: "My Title"} -removeH1: "# My Title\nContent..." → "Content..." - -// Conditional transforms -createConditionalTransform( - path => path.endsWith('README.md'), - convertH1ToTitle -) -``` - -### Step 6: Build ImportedFile Objects -- **Output**: Array of `ImportedFile` objects with: - - `sourcePath`: Original GitHub path - - `targetPath`: Final destination path - - `content`: Transformed content - - `linkContext`: Metadata for link processing - -## Phase 3: Global Link Transformation - -### Step 7: Global Link Processing -- **Function**: `globalLinkTransform()` -- **Input**: All `ImportedFile` objects -- **Configuration Used**: `linkTransform.linkMappings`, `linkTransform.stripPrefixes` - -This is the most complex phase, involving several sub-steps for each link in each file. - -#### Step 7.1: Build Source-to-Target Map -Creates a lookup table mapping original GitHub paths to final target paths: - -```javascript -sourceToTargetMap = new Map([ - ["docs/markdown/autoapi/algokit_utils/index.md", "src/content/docs/reference/algokit-utils-py/api/index.md"], - ["docs/markdown/autoapi/algokit_utils/applications/abi/index.md", "src/content/docs/reference/algokit-utils-py/api/applications/abi/index.md"] - // ... -]); -``` - -#### Step 7.2: Process Each Link in Each File - -**For each markdown link `[text](url)` in each file, the following steps occur:** - -##### Step 7.2.1: Apply Global Link Mappings -- **Configuration**: Link mappings with `global: true` -- **Purpose**: Apply transformations that should affect all links globally - -**Examples**: -```typescript -// Starlight compatibility - strips index.md for cleaner URLs -{ - pattern: /\/index(\.md)?$/, - replacement: '/', - global: true -} - -// Path prefix transformations -{ - pattern: /^docs\/code\/(.+)$/, - replacement: '/reference/algokit-utils-ts/api/$1', - global: true -} - -// Before: "../abi/index.md" -// After: "../abi/" -``` - -##### Step 7.2.2: Path Normalization -- **Function**: `normalizePath()` -- **Process**: Resolve relative paths (`../`, `./`) to full source paths - -**Examples**: -```typescript -// Current file being processed -currentFile: "docs/markdown/autoapi/algokit_utils/applications/app_factory/SendAppCreateFactoryTransactionResult.md" - -// Link after global link mappings -linkUrl: "../abi/" - -// Normalization process -currentDir = "docs/markdown/autoapi/algokit_utils/applications/app_factory" -normalized = path.posix.normalize(path.posix.join(currentDir, "../abi/")) -result = "docs/markdown/autoapi/algokit_utils/applications/abi/" -``` - -##### Step 7.2.3: Internal Link Resolution -- **Process**: Check if normalized path exists in `sourceToTargetMap` -- **If found**: Convert to Starlight URL using `pathToStarlightUrl()` -- **If not found**: Fall through to non-global link mappings - -**Examples**: -```typescript -// Success case -sourceToTargetMap.get("docs/markdown/autoapi/algokit_utils/applications/abi/index.md") -→ Found: "src/content/docs/reference/algokit-utils-py/api/applications/abi/index.md" -→ Starlight URL: "/reference/algokit-utils-py/api/applications/abi/" - -// Failure case (our current bug) -sourceToTargetMap.get("docs/markdown/autoapi/algokit_utils/applications/abi/") -→ Not found (because map has "...abi/index.md", not "...abi/") -→ Falls through to non-global link mappings -``` - -##### Step 7.2.4: Non-Global Link Mappings (Fallback) -- **Configuration**: Link mappings with `global: false` (including context-aware rules) -- **Purpose**: Handle unresolved links with context-specific transformations - -**Examples**: -```typescript -// Context-aware Python API rule -{ - contextFilter: context => context.sourcePath.startsWith('docs/markdown/autoapi/algokit_utils/'), - relativeLinks: true, - pattern: /.*/, - replacement: (match) => `/reference/algokit-utils-py/api/${match}/`, - global: false -} - -// This rule processes the already-transformed "../abi/" link -// Result: "/reference/algokit-utils-py/api/../abi/" (incorrect!) -``` - -## Phase 4: File Storage - -### Step 8: Store Processed Files -- **Function**: `storeProcessedFile()` -- **Process**: Write final content to Astro content store -- **Output**: Files available to Astro with transformed content and links - -## Configuration Reference - -### Include Patterns -```typescript -includes: [ - { - pattern: "docs/**/*.md", // Glob pattern for files to include - basePath: "src/content/docs", // Target directory - pathMappings: { // Path transformations - "docs/": "", - "docs/README.md": "overview.md" - }, - transforms: [ // Content transformations - convertH1ToTitle, - removeH1 - ] - } -] -``` - -### Link Transformations -```typescript -linkTransform: { - stripPrefixes: ['src/content/docs'], // Prefixes to remove from final URLs - linkMappings: [ - // Global link mappings (applied first) - { - pattern: /\/index\.md$/, - replacement: '/', - global: true - }, - - // Context-aware link mappings (applied to unresolved links) - { - contextFilter: context => context.sourcePath.startsWith('docs/api/'), - relativeLinks: true, - pattern: /.*/, - replacement: (match) => `/api/${match}/`, - global: false - } - ] -} -``` - -## Common Issues and Debugging - -### Link Resolution Problems - -**Problem**: Links are not being transformed correctly or are missing. - -**Debug Steps**: -1. Enable debug logging: `logLevel: 'debug'` -2. Check `[normalizePath]` debug messages to see path resolution -3. Verify your `sourceToTargetMap` includes the expected paths -4. Check if global link mappings are interfering with path resolution - -**Example Debug Output**: -``` -[normalizePath] BEFORE: linkPath="../abi/index.md", currentFilePath="docs/markdown/autoapi/algokit_utils/applications/app_factory/SendAppCreateFactoryTransactionResult.md" -[normalizePath] RELATIVE PATH RESOLVED: "../abi/index.md" -> "docs/markdown/autoapi/algokit_utils/applications/abi/index.md" -``` - -### Path Mapping Issues - -**Problem**: Files are not being placed in the correct target directories. - -**Common Causes**: -- Incorrect glob patterns in `includes[].pattern` -- Missing or incorrect `pathMappings` -- Path mapping conflicts between different include patterns - -### Asset Processing Issues - -**Problem**: Images or other assets are not being downloaded or referenced correctly. - -**Requirements**: -- Both `assetsPath` and `assetsBaseUrl` must be configured -- Asset file extensions must be included in `assetPatterns` (or use defaults) -- Local file system permissions must allow writing to `assetsPath` - -## Performance Considerations - -- **File Discovery**: Large repositories with many files will take longer to scan -- **Asset Processing**: Images and other assets are downloaded during processing -- **Link Resolution**: Complex link mapping configurations can slow down processing -- **Concurrent Processing**: The loader processes multiple configurations sequentially to avoid overwhelming GitHub's API - -## Next Steps - -For implementation details and API reference, see the main [README.md](README.md). - -For specific configuration examples, see the `examples/` directory. \ No newline at end of file diff --git a/packages/astro-github-loader/package.json b/packages/astro-github-loader/package.json index 3daf762..5105738 100644 --- a/packages/astro-github-loader/package.json +++ b/packages/astro-github-loader/package.json @@ -1,7 +1,7 @@ { "name": "@larkiny/astro-github-loader", "type": "module", - "version": "0.12.0", + "version": "0.13.0", "description": "Load content from GitHub repositories into Astro content collections with asset management and content transformations", "keywords": [ "astro", diff --git a/packages/astro-github-loader/src/github.dryrun.spec.ts b/packages/astro-github-loader/src/github.dryrun.spec.ts index e698456..a93ee9f 100644 --- a/packages/astro-github-loader/src/github.dryrun.spec.ts +++ b/packages/astro-github-loader/src/github.dryrun.spec.ts @@ -51,6 +51,18 @@ describe("github.dryrun", () => { expect(id).toBe("algorand/docs@main"); }); + it("should use custom stateKey when provided", () => { + const config: ImportOptions = { + owner: "algorandfoundation", + repo: "puya", + ref: "devportal", + stateKey: "puya-legacy-guides", + includes: [], + }; + + expect(createConfigId(config)).toBe("puya-legacy-guides"); + }); + it("should handle different refs correctly", () => { const config: ImportOptions = { name: "Test Repo", diff --git a/packages/astro-github-loader/src/github.dryrun.ts b/packages/astro-github-loader/src/github.dryrun.ts index 117c2ec..ffdb952 100644 --- a/packages/astro-github-loader/src/github.dryrun.ts +++ b/packages/astro-github-loader/src/github.dryrun.ts @@ -57,6 +57,9 @@ export interface RepositoryChangeInfo { * Creates a unique identifier for an import configuration */ export function createConfigId(config: ImportOptions): string { + if (config.stateKey) { + return config.stateKey; + } return `${config.owner}/${config.repo}@${config.ref || "main"}`; } diff --git a/packages/astro-github-loader/src/github.types.ts b/packages/astro-github-loader/src/github.types.ts index 69f5a2e..0295c40 100644 --- a/packages/astro-github-loader/src/github.types.ts +++ b/packages/astro-github-loader/src/github.types.ts @@ -271,6 +271,12 @@ export type ImportOptions = { * Display name for this configuration (used in logging) */ name?: string; + /** + * Custom state key for import tracking. When provided, overrides the default + * `owner/repo@ref` key used to track import state. This allows the same repo + * to be imported independently to multiple locations. + */ + stateKey?: string; /** * Repository owner */