Thanks so much for providing a means to mirror the PyPI repository!
After our latest run of bandersnatch mirror followed by bandersnatch verify --delete --json-update, our mirror is 13.3 TB is size. It was 17.7 TB before we ran the verify --delete operation. We found that some packages were not being updated after many runs of bandersnatch mirror. One such package was poetry. We got it to update with bandersnatch sync poetry before we ran the verify --delete operation.
We are running bandersnatch 6.3.0 and python 3.5.8 and the latest verify operation took 17 days to complete and had a zero exit code. Our mirror appears incomplete compared to the stats reported on pypi.org. How can we assess the completeness of our mirror?
On our local mirror, web/simple/index.html has 371694 . web/simple has 372444 directories and web/json has 357233 directories. The bandersnatch log reports that 1,049,164 files were fetched. https://pypi.org reports 498,484 projects and https://pypi.org/stats reports the total mirror size of 18.2 TB.
/etc/bandersnatch.conf:
[mirror]
directory = /mirror/sites/PyPI
json = true
release-files = true
cleanup = true
master = https://pypi.org
timeout = 20
global-timeout = 1800
workers = 3
hash-index = false
simple-format = ALL
stop-on-error = false
storage-backend = filesystem
verifiers = 3
compare-method = hash
Thanks so much for providing a means to mirror the PyPI repository!
After our latest run of
bandersnatch mirrorfollowed bybandersnatch verify --delete --json-update, our mirror is 13.3 TB is size. It was 17.7 TB before we ran theverify --deleteoperation. We found that some packages were not being updated after many runs ofbandersnatch mirror. One such package was poetry. We got it to update withbandersnatch sync poetrybefore we ran theverify --deleteoperation.We are running bandersnatch 6.3.0 and python 3.5.8 and the latest verify operation took 17 days to complete and had a zero exit code. Our mirror appears incomplete compared to the stats reported on pypi.org. How can we assess the completeness of our mirror?
On our local mirror, web/simple/index.html has 371694 . web/simple has 372444 directories and web/json has 357233 directories. The bandersnatch log reports that 1,049,164 files were fetched. https://pypi.org reports 498,484 projects and https://pypi.org/stats reports the total mirror size of 18.2 TB.
/etc/bandersnatch.conf: