Skip to content

perf: optimize craft get with repo/tree caching and parallel resolution#32

Merged
erdemtuna merged 1 commit into
mainfrom
feature/improve-process-speed
Mar 10, 2026
Merged

perf: optimize craft get with repo/tree caching and parallel resolution#32
erdemtuna merged 1 commit into
mainfrom
feature/improve-process-speed

Conversation

@erdemtuna

Copy link
Copy Markdown
Owner

Summary

Optimizes craft get performance with three complementary changes that together yield an estimated 7-10× speedup for typical multi-dependency workloads.

Changes

1. Repository object caching (internal/fetch/gogit.go)

  • GoGitFetcher now caches *git.Repository objects in memory
  • Previously, every ResolveRef, ReadFiles, and ListTree call triggered a full ensureRepo() cycle: acquire file lock → open repo → git fetch → return → release lock
  • For a single dependency, this happened 7-9 times; now it happens once
  • Includes double-checked locking for thread safety and memory clone deduplication

2. ListTree result caching (internal/fetch/gogit.go)

  • ListTree() results are cached by (url, commitSHA) key
  • Eliminates the redundant tree walk between resolution phase (integrity computation) and install phase (file collection)
  • Returns defensive copies to prevent caller mutation of cached data

3. Parallel dependency resolution (internal/resolve/resolver.go)

  • Phase 4 resolveOne() loop now uses errgroup.Group for concurrent execution
  • Each resolveOne() call is independent (different dep, read-only options, no shared mutable state)
  • Per-repo file locks in the fetcher layer protect against concurrent cache corruption
  • Added golang.org/x/sync dependency

Performance Impact

Metric Before After
ensureRepo() calls per dep 7-9 (each with lock+open+fetch) 1
Dep resolution Sequential Parallel via errgroup
ListTree per dep 2× (resolution + install) 1× (cached)
Estimated speedup (5 deps) baseline 7-10×

Testing

  • All existing tests pass
  • Race detector clean (go test -race ./...)
  • Pre-commit and pre-push hooks pass (vet, lint, tests)

- Cache *git.Repository objects in GoGitFetcher to eliminate redundant
  lock+open+fetch cycles (7-9× per dep reduced to 1×)
- Cache ListTree results to avoid redundant tree walks between
  resolution and install phases
- Parallelize Phase 4 (resolveOne loop) with errgroup so independent
  dependencies resolve concurrently
- Add defensive copy for cached tree slices to prevent mutation
- Cache memory clones for nil-cache path to deduplicate concurrent requests

Estimated 7-10× speedup for typical multi-dependency workloads.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@erdemtuna erdemtuna merged commit 508b818 into main Mar 10, 2026
3 checks passed
@erdemtuna erdemtuna deleted the feature/improve-process-speed branch March 29, 2026 16:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant