Skip to content

Refactor to use go-git instead of shelling out to git #19

Description

@scottbrown

Summary

Replace the os/exec calls to a system git binary with the pure-Go go-git library, so GitGrab has no runtime dependency on an external git executable.

Today GitGrab is a Go program that shells out to git for every clone/pull/fetch. That means any environment running GitGrab must also ship a full git install (and its transitive shared libraries). For a CGO_ENABLED=0 static binary this is an unnecessary and painful dependency — most acutely when packaging GitGrab into a minimal container image.

Motivation

GitGrab is consumed by KOHO's Hopper repo-sync task, which wants to run it from a minimal (scratch / distroless) container. Because GitGrab requires a real git at runtime, that image can't be truly minimal:

  • scratch/distroless ship no git and no package manager, so git + git-core helpers + every transitive .so must be hand-copied via ldd scraping — fragile and high-churn (it broke repeatedly across several fixup commits).
  • glibc's DNS resolution dlopens libnss_dns.so at runtime, which ldd does not report, so HTTPS clones silently fail in a hand-assembled image.
  • The workaround is to fall back to Alpine + apk add git, which defeats the goal of a minimal, low-CVE-surface image.

A pure-Go implementation makes GitGrab a single self-contained static binary that drops cleanly into FROM scratch (plus CA certs), eliminating all of the above.

Current behavior

git is invoked via os/exec in two places:

cmd/gitgrab/main.go

  • exec.LookPath("git") — hard preflight check that aborts if git isn't on PATH.

gitgrab.go

  • getCurrentBranch()git -C <path> branch --show-current
  • CloneRepo():
    • git clone <url> <path> for new repos
    • git -C <path> pull when on the default branch (main/master)
    • git -C <path> fetch when on a non-default branch, or as a fallback when branch detection fails / default branch is unknown

Auth is handled by URL construction:

  • HTTP: private repos use https://<token>@github.com/<org>/<repo>.git
  • SSH: git@github.com:<org>/<repo>.git (relies on the host's SSH agent/keys)

Proposed change

Swap the subprocess calls for go-git (github.com/go-git/go-git/v5), keeping the existing CLI, flags, and behavior identical.

Rough operation mapping:

Current subprocess go-git equivalent
git clone <url> <path> git.PlainClone(path, false, &git.CloneOptions{URL, Auth})
git branch --show-current repo.Head()ref.Name().Short()
git pull worktree.Pull(&git.PullOptions{Auth, ...}), treating git.NoErrAlreadyUpToDate as success
git fetch repo.Fetch(&git.FetchOptions{Auth, ...}), treating git.NoErrAlreadyUpToDate as success
open existing repo git.PlainOpen(path)

Auth:

  • HTTP/token: &githttp.BasicAuth{Username: "x-access-token", Password: <token>} instead of embedding the token in the URL (also keeps the token out of any URL that might get logged).
  • SSH: go-git's ssh transport (gitssh.NewSSHAgentAuth or NewPublicKeysFromFile) with an explicit known-hosts / host-key callback policy.

Also remove the exec.LookPath("git") preflight in main.go — it becomes meaningless once there's no external git.

Design considerations & edge cases

  • SSH host-key verification: the shelled-out git relied on the host's known_hosts. With go-git we must decide the HostKeyCallback policy explicitly. Container use will likely need known-hosts seeded or a documented policy.
  • pull semantics: go-git Pull fast-forwards the current branch; ensure divergent/non-fast-forward cases are handled the same way the CLI currently tolerates them (surface as a per-repo failure, keep going).
  • git.NoErrAlreadyUpToDate: both Pull and Fetch return this sentinel when there's nothing to do — must be treated as success, not an error, to preserve current output.
  • Shallow clones: if we want to keep sync fast/cheap, evaluate CloneOptions.Depth (note go-git's historical rough edges with shallow fetch).
  • Git LFS: go-git does not support LFS. Confirm none of the target repos rely on LFS, or document the limitation.
  • Submodules: default behavior differs; set SubmoduleRescursivity/recurse options intentionally if submodules matter.
  • Performance/memory: go-git can use more memory and be slower than C git on very large repos/monorepos — worth a sanity check against the largest KOHO repos.
  • CGO: go-git is pure Go, so CGO_ENABLED=0 static builds continue to work (and now actually deliver a dependency-free binary).

Acceptance criteria

  • No os/exec usage remains; GitGrab runs with no git binary on PATH.
  • gitgrab -o <org> -m http <dir> clones new repos and updates existing ones (pull on default branch, fetch otherwise), matching current behavior.
  • gitgrab -o <org> -m ssh <dir> works with a documented host-key policy.
  • "Already up to date" is reported as success.
  • CLI flags, output format, and exit codes are unchanged.
  • Builds as a static binary (CGO_ENABLED=0) and runs in a FROM scratch image with only CA certs added.

Testing

  • Unit tests around branch detection and the clone/pull/fetch decision logic (the existing HTTPClient interface pattern makes the GitHub API side already testable; do the same for the git operations behind an interface so they can be faked).
  • Integration test against a throwaway org (or a local bare repo fixture) exercising: fresh clone, no-op update, update-with-changes, non-default-branch fetch.
  • Verify the resulting binary runs in scratch.

Out of scope

  • Changing the CLI surface, flags, or auth model.
  • Adding LFS support.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions