Repository navigation
feat(profile): migration summary, duration estimate, and external-tool signals - #25
Merged
Merged
Conversation
Adds an org-level "Migration summary" to the profile detail page: a
preparation checklist (which external tooling/reconfiguration the
migration needs, and how many repos each touches) plus a coarse,
size-band duration estimate with adjustable parallelism.
The batched augment pass now also gathers each repo's GitHub Packages
count and Git LFS usage (root .gitattributes filter=lfs), lighting up
the packages and git-lfs considerations that were previously
indeterminate.
- augment: + packages{totalCount} and a .gitattributes blob lookup
- analyze: + packages and git-lfs detectors
- summary.ts: buildPreparationSummary rolls per-repo findings up by
consideration, joins registry remediation metadata, counts affected
repos, and honestly lists not-yet-crawled considerations
- estimate.ts: estimateDuration buckets repos into S/M/L/XL size bands
(tunable hour ranges) and sums sequential work; wall-clock is
total / parallelism, divided client-side so the knob stays live
- format: + formatHours for coarse duration display
- service: ProfileDetail gains { summary, estimate }
- UI: Migration summary section (readiness rollup, estimate + parallel
input, prep checklist, coverage note)
Estimate constants are rough and centralized for calibration. The new
GraphQL fields need validation against a live org.
11 gates green; coverage 87.23/85.42.
The batched augment query now also scans release assets (first 100 releases x first 50 assets each) and sums their bytes into RepoSignals.releaseAssetBytes. Two blockers light up: - release-size-limit: >= 10 GiB of assets (GHEC's per-repo release cap) - metadata-archive-limit: >= 40 GiB (the Importer's metadata limit, usually driven by binary release assets) Bounded scan, so it's an estimate (the registry already marks these estimated); summing is monotonic, so truncation only risks a false negative just under a threshold, never a false positive.
GraphQL doesn't expose rulesets, so the runner makes one best-effort
REST call per run (GET /orgs/{org}/rulesets) and persists the count on
the run (profile_runs.org_ruleset_count, additive column). Org rulesets
aren't migrated and certain ones can fail the migration outright, so the
profile detail page surfaces a callout when any exist.
- core/auth: getSourceClients() exposes a source REST client (replaces
getSourceGraphql, which only the profiler used)
- profile/rulesets: getOrgRulesetCount(rest, org), 0 on any error
- runner: injectable getOrgRulesetCount dep (default 0; service binds
the REST-backed impl), runProfile now merges Partial deps
- store/schema: profile_runs.org_ruleset_count + setProfileRunRulesets
- ProfileRun gains orgRulesetCount; UI shows an org-ruleset warning
The rulesets endpoint needs validation against a live org.
11 gates green; coverage 87.34/85.74.
Adds one best-effort REST pass per run for the org-scoped resources that
aren't migrated and must be recreated on the target:
- Actions secrets & variables (/orgs/{org}/actions/secrets|variables)
- Dependabot secrets (/orgs/{org}/dependabot/secrets)
- Codespaces secrets (/orgs/{org}/codespaces/secrets)
- Self-hosted runners (/orgs/{org}/actions/runners)
- Custom property definitions (/orgs/{org}/properties/schema)
Counts only — secret values are never exposed by these endpoints. The
six calls run concurrently and each degrades to 0 on a missing scope,
unavailable endpoint, or network error, so the crawl never fails on them.
Persisted as a JSON column profile_runs.org_resources (additive) and
surfaced in the Migration summary as an 'Organization-level resources to
recreate' block (non-zero items only). Gathered concurrently with the
org rulesets pass in the runner.
These are org-scoped, so repo-scoped copies of the same resources are
not covered — that needs the per-repo REST pass (deferred).
11 gates green; coverage 87.76/86.40.
Folds a .github/workflows tree lookup into the existing batched augment query (object(expression: "HEAD:.github/workflows")), counting .yml/.yaml definition files into RepoSignals.workflowFileCount. Lights up the actions-run-history consideration per repo: workflows migrate, but their run history and artifacts do not, so any workflow file means there's history that will be lost. Near-zero cost — it rides the same per-repo GraphQL alias as the other signals, no extra requests. DETECTED considerations now 13. 11 gates green; coverage 87.76/86.40.
Two batching optimizations to get more signal from fewer, faster calls.
1. Release-partition. The release-asset scan (up to 100 releases x 50
assets per repo, ~5k nodes) was the single field forcing the augment
batch down to 25 repos/request. Discovery now also pulls the free
releases.totalCount on its 100-wide page, so the runner splits repos:
those with zero releases skip the asset scan entirely and batch at 90
per request; only repos that actually have releases stay at 25. For a
typical org (most repos have no releases) this cuts the augment leg of
the crawl ~3x.
- DiscoveredRepo gains releasesCount (was a RepoSignals/augment field)
- augmentRepoSignals takes { scanReleases }; the query omits the
release-asset selection when false
2. Concurrent augment chunks. The chunk loop was sequential. A bounded
worker pool (3 in flight) now processes chunks concurrently, cutting
wall-clock ~3x without stressing rate limits (the throttling plugin
still guards secondary limits). Completed chunks persist regardless;
the first chunk error fails the run after in-flight chunks settle
(same stop-on-first-failure semantics as before).
Request budget for N repos is now ceil(N/100) discover + the partitioned
augment + 7 org-level, run with concurrency. Primary rate cost is ~1
point/request against 5k/hr; the real constraint is the per-query node
budget + 10s timeout, which the partition respects.
11 gates green; coverage 87.76/86.40.
This was referenced Jun 15, 2026
fix(profile): reliable org crawl (REST discovery + two-pass GraphQL) plus repo search/pagination
#26
Merged
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds an org-level Migration summary to the profile detail page — a preparation checklist (what external tooling/reconfiguration the migration needs, and how many repos each touches) plus a coarse duration estimate with adjustable parallelism — and gathers the remaining signals needed to surface the named considerations.
Three commits, each green:
1. Migration summary + duration estimate (
f0a58af)summary.ts): rolls per-repo findings up by consideration, joins each one's external-tool remediation from the registry, counts affected repos, and honestly lists what's not yet crawled. Plus repo-level blocker/warn/clean tallies.estimate.ts): buckets repos into S/M/L/XL size bands (tunable hour ranges) and sums sequential work. Wall-clock = total ÷ parallelism, divided client-side so the parallelism knob (default 10, the migrate queue cap) stays live..gitattributesfilter=lfs), lighting up two previously-indeterminate considerations.format.ts:formatHoursfor coarse duration display.2. Oversized release assets (
2810c1e)releaseAssetBytes. Two blockers light up:release-size-limit(≥ 10 GiB, GHEC's per-repo release cap) andmetadata-archive-limit(≥ 40 GiB). Bounded → an estimate; summing is monotonic so truncation only risks a false negative just under a threshold.3. Org rulesets (
512d873)GET /orgs/{org}/rulesets) and persists the count (profile_runs.org_ruleset_count, additive column). The page shows a warning callout when any exist — an org ruleset can fail the migration outright.Named items, now covered
.gitattributes)Estimate model
Deliberately coarse per the chosen approach: size bands × parallelism, with centralized, tunable constants clearly labeled "rough estimate — calibrate as real timings accumulate." There's no published Importer throughput figure, so this is a transparent heuristic, not a guarantee.
Verification
bun run verify:json— all 11 gates green; coverage 87.34 / 85.74Needs live validation
The new GraphQL fields (
packages,gitattributesblob, release assets) and the rulesets REST endpoint can't be exercised against a live org from here. All are defensive (partial-data recovery; rulesets degrades to 0 on any error), but a first crawl against a real org should confirm the query shapes and token scopes.