Inspecting ports for modifications to git checkout yields 100% CPU usage and takes 30 min+

[herrbischoff]

Prerequisites

• Have you checked for an existing issue describing your problem?
• Are you running the latest version?
• Is your ports tree recent?
• Is your FreeBSD Host on a supported release?

Describe the bug

I'm running Poudriere within a jail on a small VPS with 2 CPU cores and 2 GB of RAM. I'm only looking to build a couple of packages with custom options, not operate a full build server. Whenever I'm running the bulk command, this is what happens:

[...]
[00:00:23] Inspecting /usr/local/poudriere/data/.m/FreeBSD_15_amd64-HEAD/ref//usr/ports for modifications to git checkout... no
[00:30:39] Ports top-level git hash: 439ad72612af8d38b4280cbbd3c1e3104daffd3f
[...]

It takes Poudriere over 30 min total to check for modification to the Git checkout and consumes almost 100% of CPU while doing so. I remember it taking a bit but this is excessive and has to be a bug. It takes longer for Pourdriere to check for Git changes than the entire following build process.

How to reproduce

Steps to reproduce the behavior: run poudriere bulk -j FreeBSD:15:amd64 -p HEAD -b latest -f /usr/local/etc/poudriere.d/port-list on my machine.

Expected behavior

Checking for modifications taking a lot less time.

Screenshots

See log excerpt above.

Environment

• Host OS [e.g. 12.2 amd64]: 15.0-RELEASE-p5
• Jail OS [e.g. 12.0 powerpc]: 15.0-RELEASE-p5
• Browser: [e.g. chrome, safari]: n/a
• Poudriere Version [e.g. 3.3.1 or git hash or port version]: poudriere-git-3.4.99.20260415
• Ports branch and revision [e.g. 2020Q3 r550754]: main 439ad72612

[bdrewery]

I have some prototype code to make that check asynchronous, but it wouldn't help much if it is taking 30 minutes.

You can disable it entirely with GIT_TREE_DIRTY_CHECK=no in your poudriere.conf.

[herrbischoff]

You can disable it entirely with GIT_TREE_DIRTY_CHECK=no in your poudriere.conf.

Thanks for this, it does help. Still, that the process takes 30 minutes and longer out of the box on a lower-end instance does hint at something rather extremely inefficient within the loop – Git, Poudriere itself, or both. The instance does use NVMe storage and random access speeds are pretty fast, so it's not likely to be constrained IO.

[bdrewery]

It is mostly this:

cd $PORTSDIR
git \
    -c core.checkStat=minimal \
    -c core.fileMode=off \
    -c status.renames=false \
    -c core.untrackedCache=true \
    -c advice.statusUoption=false \
    status \
    --ignored \
    --porcelain .

The output is then parsed through | awk -f /usr/local/share/poudriere/awk/git_dirty.awk.

[herrbischoff]

Thanks for the details. This is interesting. Running the commands manually within/usr/local/poudriere/ports/HEAD completes within a very reasonable amount of time.

root@poudriere:/usr/local/poudriere/ports/HEAD # time git \
    -c core.checkStat=minimal \
    -c core.fileMode=off \
    -c status.renames=false \
    -c core.untrackedCache=true \
    -c advice.statusUoption=false \
    status \
    --ignored \
    --porcelain . | awk -f /usr/local/share/poudriere/awk/git_dirty.awk
       18.76 real         0.77 user        11.07 sys

Poudriere's mount path of /usr/local/poudriere/data/.m/FreeBSD_15_amd64-HEAD/ref//usr/ports within the bulk process still takes about half an hour.

[bdrewery]

@herrbischoff Can you try it in /usr/local/poudriere/ports/HEAD after a fresh reboot and let me know?

[bdrewery]

Poudriere does process the data into shared files. That could be slow but not half an hour slow as the same data caching is done for ports metadata retrieval and writes a lot more.

[bdrewery]

Tell me more about the storage inside the VM and outside? So I can repro the case locally.

[herrbischoff]

After a full reboot of the machine, it's even quicker:

root@poudriere:/usr/local/poudriere/ports/HEAD # time git \
    -c core.checkStat=minimal \
    -c core.fileMode=off \
    -c status.renames=false \
    -c core.untrackedCache=true \
    -c advice.statusUoption=false \
    status \
    --ignored \
    --porcelain . | awk -f /usr/local/share/poudriere/awk/git_dirty.awk
        9.19 real         0.60 user         4.53 sys

The bulk command (again) ran for so long that I aborted it to measure the below info.

The environment is within a Bastille-created jail. The filesystem is ZFS with zstd compression. The Poudriere config therefore does not use ZFS. I've tried lz4 compression in place of zstd, just to be sure, but it doesn't make a difference.

The VPS itself it a Hetzner CPX11 instance (https://www.hetzner.com/cloud/regular-performance) using NVMe SSD storage.

Output of diskinfo -c /dev/da0:

/dev/da0
	512         	# sectorsize
	40961572864 	# mediasize in bytes (38G)
	80003072    	# mediasize in sectors
	4096        	# stripesize
	0           	# stripeoffset
	4979        	# Cylinders according to firmware.
	255         	# Heads according to firmware.
	63          	# Sectors according to firmware.
	QEMU QEMU HARDDISK	# Disk descr.
	            	# Disk ident.
	vtscsi0     	# Attachment
	Yes         	# TRIM/UNMAP support
	0           	# Rotation rate in RPM
	Not_Zoned   	# Zone Mode

I/O command overhead:
	time to read 10MB block      0.012406 sec	=    0.001 msec/sector
	time to read 20480 sectors   4.638851 sec	=    0.227 msec/sector
	calculated command overhead			=    0.226 msec/sector

Output of diskinfo -t /dev/da0:

/dev/da0
	512         	# sectorsize
	40961572864 	# mediasize in bytes (38G)
	80003072    	# mediasize in sectors
	4096        	# stripesize
	0           	# stripeoffset
	4979        	# Cylinders according to firmware.
	255         	# Heads according to firmware.
	63          	# Sectors according to firmware.
	QEMU QEMU HARDDISK	# Disk descr.
	            	# Disk ident.
	vtscsi0     	# Attachment
	Yes         	# TRIM/UNMAP support
	0           	# Rotation rate in RPM
	Not_Zoned   	# Zone Mode

Seek times:
	Full stroke:	  250 iter in   0.083629 sec =    0.335 msec
	Half stroke:	  250 iter in   0.090283 sec =    0.361 msec
	Quarter stroke:	  500 iter in   0.181255 sec =    0.363 msec
	Short forward:	  400 iter in   0.131302 sec =    0.328 msec
	Short backward:	  400 iter in   0.146896 sec =    0.367 msec
	Seq outer:	 2048 iter in   0.511533 sec =    0.250 msec
	Seq inner:	 2048 iter in   0.552675 sec =    0.270 msec

Transfer rates:
	outside:       102400 kbytes in   0.129251 sec =   792257 kbytes/sec
	middle:        102400 kbytes in   0.124344 sec =   823522 kbytes/sec
	inside:        102400 kbytes in   0.056225 sec =  1821254 kbytes/sec

Let me know if you need something else.

[bdrewery]

There is a real .git/ dir at /usr/local/poudriere/ports/HEAD/.git/ or it is higher up in /usr/local/poudriere somewhere?

[herrbischoff]

There is a real .git/ dir at /usr/local/poudriere/ports/HEAD/.git/ or it is higher up in /usr/local/poudriere somewhere?

Yes, it is.

root@poudriere:/usr/local/poudriere/ports/HEAD # ls -lah
total 3.0 MB
drwxr-xr-x    70 root wheel   82B Apr 27 18:33 .
drwxr-xr-x     3 root wheel    3B Jul 28  2025 ..
-rw-r--r--     1 root wheel   56B Jul 28  2025 .arcconfig
drwxr-xr-x     7 root wheel   15B Apr 28 08:54 .git
-rw-r--r--     1 root wheel  178B Sep  3  2025 .gitignore
drwxr-xr-x     3 root wheel    5B Jul 28  2025 .hooks
-rw-r--r--     1 root wheel  230B Mar 21 13:32 .mailmap
-rw-r--r--     1 root wheel  145K Mar 21 13:32 CHANGES
-rw-r--r--     1 root wheel  727B Jul 28  2025 CONTRIBUTING.md
-rw-r--r--     1 root wheel  1.4K Mar 21 13:32 COPYRIGHT
-rw-r--r--     1 root wheel   13K Apr 25 10:06 GIDs
drwxr-xr-x     2 root wheel   23B Mar 21 13:32 Keywords
-rw-r--r--     1 root wheel  513K Apr 27 18:33 MOVED
-rw-r--r--     1 root wheel  7.1K Jul 28  2025 Makefile
drwxr-xr-x     6 root wheel   25B Apr 27 18:33 Mk
-rw-r--r--     1 root wheel  1.2K Jul 28  2025 README
drwxr-xr-x     3 root wheel   12B Mar 21 13:32 Templates
drwxr-xr-x     3 root wheel    5B Jul 28  2025 Tools
-rw-r--r--     1 root wheel   48K Apr 25 10:06 UIDs
-rw-r--r--     1 root wheel  177K Apr 26 09:10 UPDATING
drwxr-xr-x    39 root wheel   40B Apr 25 10:06 accessibility
drwxr-xr-x    11 root wheel   13B Jul 28  2025 arabic
drwxr-xr-x   308 root wheel  309B Apr 25 10:06 archivers
drwxr-xr-x   156 root wheel  157B Mar 21 13:32 astro
[...]