Does ghorg also make a copy of objects stored in git LFS?

[dotboris]

I'm looking into a tool for backing up my GitHub repos and some of those use Git LFS to store large files. ghorg looks like a great solution for this but I have one question. I'm wondering if ghorg will include LFS objects in its clones.

[gabrie30]

Ghorg will use git that is installed locally. Just reading the git lfs docs it seems like as long as you have the extension locally and your repo is setup for lfs it should work how you expect. Let me know if thats not the case.

[dotboris]

I've finally gotten around to testing this and I can't see to get it to work.

I'm running:

ghorg clone dotboris --scm github -c user --path ./local-backup --backup --token "$(gh auth token)" --skip-forks --skip-archived

I'm using my prod-guard repo which has images stored in LFS: https://github.com/dotboris/prod-guard/tree/main/doc

Looking at the content of the backed up directory, I get:

$ ls -lah local-backup/dotboris_backup/prod-guard/
Permissions Size User     Date Modified Name
drwxr-xr-x     - dotboris 15 Dec 14:11  branches
.rw-r--r--   173 dotboris 15 Dec 14:12  config
.rw-r--r--    73 dotboris 15 Dec 14:11  description
.rw-r--r--  141k dotboris 15 Dec 14:12  FETCH_HEAD
.rw-r--r--    21 dotboris 15 Dec 14:11  HEAD
drwxr-xr-x     - dotboris 15 Dec 14:11  hooks
drwxr-xr-x     - dotboris 15 Dec 14:11  info
drwxr-xr-x     - dotboris 15 Dec 14:12  lfs
drwxr-xr-x     - dotboris 15 Dec 14:12  objects
.rw-r--r--   72k dotboris 15 Dec 14:11  packed-refs
drwxr-xr-x     - dotboris 15 Dec 14:11  refs

I get an LFS directory which is promising but it's actually empty. If I try to clone from that backed up repository, I also get this error:

$ git clone ./local-backup/dotboris_backup/prod-guard/ prod-guard
Cloning into 'prod-guard'...
done.
Downloading doc/border-showcase.png (93 KB)
Error downloading object: doc/border-showcase.png (1d0e0f3): Smudge error: Error downloading doc/border-showcase.png (1d0e0f36c831a0ce57748e2fdb2e373ecb49baaa367d44390fa6d313797e2864): error transferring "1d0e0f36c831a0ce57748e2fdb2e373ecb49baaa367d44390fa6d313797e2864": [0] remote missing object 1d0e0f36c831a0ce57748e2fdb2e373ecb49baaa367d44390fa6d313797e2864

Errors logged to '/home/dotboris/code/homelab/__scratch/prod-guard/.git/lfs/logs/20241215T141901.396453395.log'.
Use `git lfs logs last` to view the log.
error: external filter 'git-lfs filter-process' failed
fatal: doc/border-showcase.png: smudge filter lfs failed
warning: Clone succeeded, but checkout failed.
You can inspect what was checked out with 'git status'
and retry with 'git restore --source=HEAD :/'

Here's the error mentioned in that error:

git lfs error log

cat /home/dotboris/code/homelab/__scratch/prod-guard/.git/lfs/logs/20241215T141901.396453395.log
git-lfs/3.5.1 (3.5.1; linux amd64; go 1.22.7)
git version 2.46.0

$ git-lfs filter-process
Error downloading object: doc/border-showcase.png (1d0e0f3): Smudge error: Error downloading doc/border-showcase.png (1d0e0f36c831a0ce57748e2fdb2e373ecb49baaa367d44390fa6d313797e2864): error transferring "1d0e0f36c831a0ce57748e2fdb2e373ecb49baaa367d44390fa6d313797e2864": [0] remote missing object 1d0e0f36c831a0ce57748e2fdb2e373ecb49baaa367d44390fa6d313797e2864

error transferring "1d0e0f36c831a0ce57748e2fdb2e373ecb49baaa367d44390fa6d313797e2864": [0] remote missing object 1d0e0f36c831a0ce57748e2fdb2e373ecb49baaa367d44390fa6d313797e2864
github.com/git-lfs/git-lfs/v3/errors.New
	github.com/git-lfs/git-lfs/v3/errors/errors.go:62
github.com/git-lfs/git-lfs/v3/tq.(*customAdapter).DoTransfer
	github.com/git-lfs/git-lfs/v3/tq/custom.go:311
github.com/git-lfs/git-lfs/v3/tq.(*adapterBase).worker
	github.com/git-lfs/git-lfs/v3/tq/adapterbase.go:183
runtime.goexit
	runtime/asm_amd64.s:1695
Error downloading doc/border-showcase.png (1d0e0f36c831a0ce57748e2fdb2e373ecb49baaa367d44390fa6d313797e2864)
github.com/git-lfs/git-lfs/v3/errors.newWrappedError
	github.com/git-lfs/git-lfs/v3/errors/types.go:229
github.com/git-lfs/git-lfs/v3/errors.Wrapf
	github.com/git-lfs/git-lfs/v3/errors/errors.go:85
github.com/git-lfs/git-lfs/v3/lfs.(*GitFilter).downloadFile
	github.com/git-lfs/git-lfs/v3/lfs/gitfilter_smudge.go:134
github.com/git-lfs/git-lfs/v3/lfs.(*GitFilter).Smudge
	github.com/git-lfs/git-lfs/v3/lfs/gitfilter_smudge.go:85
github.com/git-lfs/git-lfs/v3/commands.smudge
	github.com/git-lfs/git-lfs/v3/commands/command_smudge.go:128
github.com/git-lfs/git-lfs/v3/commands.filterCommand
	github.com/git-lfs/git-lfs/v3/commands/command_filter_process.go:131
github.com/spf13/cobra.(*Command).execute
	github.com/spf13/cobra@v1.7.0/command.go:944
github.com/spf13/cobra.(*Command).ExecuteC
	github.com/spf13/cobra@v1.7.0/command.go:1068
github.com/spf13/cobra.(*Command).Execute
	github.com/spf13/cobra@v1.7.0/command.go:992
github.com/git-lfs/git-lfs/v3/commands.Run
	github.com/git-lfs/git-lfs/v3/commands/run.go:154
main.main
	github.com/git-lfs/git-lfs/v3/git-lfs.go:34
runtime.main
	runtime/proc.go:271
runtime.goexit
	runtime/asm_amd64.s:1695
Smudge error
github.com/git-lfs/git-lfs/v3/errors.newWrappedError
	github.com/git-lfs/git-lfs/v3/errors/types.go:229
github.com/git-lfs/git-lfs/v3/errors.NewSmudgeError
	github.com/git-lfs/git-lfs/v3/errors/types.go:315
github.com/git-lfs/git-lfs/v3/lfs.(*GitFilter).Smudge
	github.com/git-lfs/git-lfs/v3/lfs/gitfilter_smudge.go:102
github.com/git-lfs/git-lfs/v3/commands.smudge
	github.com/git-lfs/git-lfs/v3/commands/command_smudge.go:128
github.com/git-lfs/git-lfs/v3/commands.filterCommand
	github.com/git-lfs/git-lfs/v3/commands/command_filter_process.go:131
github.com/spf13/cobra.(*Command).execute
	github.com/spf13/cobra@v1.7.0/command.go:944
github.com/spf13/cobra.(*Command).ExecuteC
	github.com/spf13/cobra@v1.7.0/command.go:1068
github.com/spf13/cobra.(*Command).Execute
	github.com/spf13/cobra@v1.7.0/command.go:992
github.com/git-lfs/git-lfs/v3/commands.Run
	github.com/git-lfs/git-lfs/v3/commands/run.go:154
main.main
	github.com/git-lfs/git-lfs/v3/git-lfs.go:34
runtime.main
	runtime/proc.go:271
runtime.goexit
	runtime/asm_amd64.s:1695

Current time in UTC:
2024-12-15 19:19:01

Environment:
LocalWorkingDir=/home/dotboris/code/homelab/__scratch/prod-guard
LocalGitDir=/home/dotboris/code/homelab/__scratch/prod-guard/.git
LocalGitStorageDir=/home/dotboris/code/homelab/__scratch/prod-guard/.git
LocalMediaDir=/home/dotboris/code/homelab/__scratch/prod-guard/.git/lfs/objects
LocalReferenceDirs=
TempDir=/home/dotboris/code/homelab/__scratch/prod-guard/.git/lfs/tmp
ConcurrentTransfers=8
TusTransfers=false
BasicTransfersOnly=false
SkipDownloadErrors=false
FetchRecentAlways=false
FetchRecentRefsDays=7
FetchRecentCommitsDays=0
FetchRecentRefsIncludeRemotes=true
PruneOffsetDays=3
PruneVerifyRemoteAlways=false
PruneVerifyUnreachableAlways=false
PruneRemoteName=origin
LfsStorageDir=/home/dotboris/code/homelab/__scratch/prod-guard/.git/lfs
AccessDownload=none
AccessUpload=none
DownloadTransfers=basic,lfs-standalone-file,ssh
UploadTransfers=basic,lfs-standalone-file,ssh
GIT_DIR=/home/dotboris/code/homelab/__scratch/prod-guard/.git
GIT_EXEC_PATH=/nix/store/0l63mqr7qx1asjxfz86j4zbs4758gn98-git-2.46.0/libexec/git-core

Client IP addresses:
(REDACTED just in case)

If I go in the backed-up repo and run git lfs fetch things start working fine:

# Running git lfs fetch
$ cd local-backup/dotboris_backup/prod-guard/
$ git lfs fetch
fetch: Fetching reference refs/heads/main

# lfs directory now has stuff in it
$ fd . lfs --type file
lfs/objects/1d/0e/1d0e0f36c831a0ce57748e2fdb2e373ecb49baaa367d44390fa6d313797e2864
lfs/objects/25/56/25563e5a667a8fe6b92a2f34ba2354724bb98688becb1b1ae9a659be7cfee58a
lfs/objects/26/f5/26f534b5be287213c6d6498b690bdc1cdb49d44f5e34d2b25e689a0bbfc16521
lfs/objects/2b/74/2b74fcd6c38eb603d9c86cd1c8cb97ba423d200d7e3e555cbc5a704ac456e00f
lfs/objects/43/dd/43dd1369e973372c8378d51484f27c58c2c864e1833667fda64a9841055b6973
lfs/objects/5a/0f/5a0f222e5f9f0ef025f969c38bee064dd05206bde1ccaca978ae136bf441b74e
lfs/objects/60/d0/60d0956ffd006144c42fb7572204e0082f42843453648ac7db9737ea443a6e69
lfs/objects/6c/f6/6cf60a11c0b07d400446df7d650c034c9cf4d87971496c71aa30b7cadb16c5f3
lfs/objects/7e/3a/7e3a71959290e14fe03978fed03a838b2ff0cbf124b2810dbe359080d8cac996
lfs/objects/81/fc/81fccf49ee281e580048db4e8f9cfe2d1c7ff1bad3a0cd558947b5779edafd1f
lfs/objects/8c/d6/8cd66c8f5d5adc8288e6e6471aecf3c3e4be9ac3d0d6a5957863b89eb560380c
lfs/objects/b6/e5/b6e5fdd45e868545779caab0e01a19cae91f5adba2a164d86b777838eca33863
lfs/objects/b7/7b/b77b168709ca08fc66c2361e793116d731f4eafcb2f38c5770d0f7abdeb445b0
lfs/objects/be/34/be34c8bf17fc7f2136db5c5bd2c684fae5d61c13d7fa18132cfb8264578c349e
lfs/objects/cd/49/cd49d6d730d5cef3c7531a3cfe27167f62eb1d8e90f4e67d25e9d8602e333245
lfs/objects/d3/4f/d34f5722da8f5353f50fb7427dc165bed7ce01a7bfb6bca2d26491300a99a260
lfs/objects/df/ac/dfaca875362b100b19b0ceb82bcdd98f3acfaacf8c1c08048dc564b62c25451b
lfs/objects/e7/83/e7837f4e271a6ad88a162d51d34051177dd07ed8a34f6c2107dbc2cf3325b3c1

# Trying to clone again
$ cd ../../..
$ rm -rf prod-guard
$ git clone ./local-backup/dotboris_backup/prod-guard/ prod-guard
Cloning into 'prod-guard'...
done.

# Testing
$ open prod-guard/doc/bottom-banner-showcase.png
# The image opens up correctly

A little bit of googling led me to discover that I should really have run git lfs fetch --all: git-lfs/git-lfs#1338

[dotboris]

It's been a while but I've finally taken the time to look into this.

First, I should mention that I am currently using ghorg and it is not copying content from git lfs. I've managed to boil it down to this:

• When you run ghorg clone ... with --backup, then LFS content is not copied at all
• When you run ghorg clone ... without --backup, then some LFS content is copied

Here's what's going on as far as I understand:

The core idea behind git LFS is that large files are not committed to the repo and instead are pushed onto a separate git LFS server. This is done through clean and smudge filters as well as a pre-push hook.

When you add files to git lfs with git lfs track ..., it adds the following to .gitattributes in your repo: (example for png files)

*.png filter=lfs diff=lfs merge=lfs -text

This tells git to the the use the lfs filter for the tracked files. This does many things but there are two that matter for us.

• When a file is committed, the lfs clean filter replaces it with a text file that holds a reference used to find the real file on the lfs server. This is why the actually file never gets committed.
• When a file is checked out, the lfs smudge filter reads the text file from above and uses the reference within to download the real file from the lfs server. It then replaces the text file with the real file on the file system.

What's important to note here is that downloading files from lfs happens at checkout time. This explains the behavior that I'm seeing. When running ghorg clone ... without --backup it'll clone the repo normally and git will do a checkout on the default branch. This triggers the filter and downloads the latest version of the files tracked by lfs. When running ghorg clone ... with --backup, ghorg runs git clone --mirror which performs a bare clone. That means that there's no checkout. The lfs filter never runs and no files get downloaded.

In my case, I'm using --backup and that's why I'm not seeing any content from lfs.

With that in mind, I think that it's fair to say that ghorg doesn't really handle lfs. When using --backup, nothing gets copied. When --backup is not being used, only the latest version is downloaded which is nice but doesn't work too well for backup purposes.

I think that it'd be great if ghorg would support LFS. From my testing and what I've read, it should be as simple as running git lfs fetch --all after a repo is cloned or after it has been updated. I'd be happy to give implementing this a shot.

[dotboris]

As of https://github.com/gabrie30/ghorg/releases/tag/v1.11.12, this is supported. You simply have to pass --fetch-git-lfs or GHORG_FETCH_GIT_LFS=true to ghorg clone ....