Image/media migration into Archetype 3

[zixben]

@geourjoa (question about importing the images into Archetype 3):
My next challenge will be to import the images into Archetype 3 iiif server. I have seen nothing on it, did you have some pointer ?

@lguariento : @sjabrookes mention me that you have already done things like that. Do you have some tips or docs for me ?

[zixben]

Image handling

For images, Archetype 3 is not really “uploading” them through Django. The important point is that the target ItemImage.image field stores a relative image identifier/path, and the IIIF server serves the corresponding file from the media filesystem.

In the current backend compose setup:

• Django media root is storage/media/;
• SIPI mounts the same folder as /sipi/images;
• IIIF_HOST points to the SIPI server, for example http://localhost:1024.

So if an imported image row has:

image = some/folder/page_001.jp2

then the file needs to exist at:

backend/storage/media/some/folder/page_001.jp2

and SIPI should expose it as:

http://localhost:1024/some%2Ffolder%2Fpage_001.jp2/info.json

The frontend/API then uses the iiif_image value from the manuscript image endpoints to render it.

Likely migration task

The image migration task is probably to:

1. decide the target relative path convention for legacy images;
2. copy or rsync the actual image files into storage/media/, preserving those relative paths;
3. make sure manuscripts_itemimage.image contains the matching relative path;
4. verify a few files directly through SIPI using /info.json.

If the importer already populated the image path values correctly, then the remaining work may just be copying the files into the mounted media directory.

If not, we need a small mapping step from the legacy image/file records to the target ItemImage.image values.

Leaving room for @lguariento’s experience here, but that is the storage/rendering path used by the current Archetype 3 stack.

[zixben]

One separate point: carousel/site images are not part of the IIIF manuscript image flow.

Carousel images are ordinary Django media files. In Archetype 3, new carousel uploads go under:

backend/storage/media/carousel/

and the database stores a media-relative path in publications_carouselitem.image, for example:

carousel/example.jpg

The site then renders it via Django media as:

/media/carousel/example.jpg

So for migrated carousel images, either:

1. preserve the legacy media-relative path and copy the files under the matching storage/media/... location; or
2. normalize them to carousel/... and update the imported CarouselItem.image values accordingly.

The key rule is simply:

The database path must match the file path under MEDIA_ROOT.

This is separate from manuscript images, where ItemImage.image is treated as an IIIF identifier/path and served through SIPI.

Also worth noting: this checkout currently has both:

storage/media/carousel/

and:

storage/media/uploads/Carousel/

The current code-created carousel uploads use lowercase carousel/, but imported legacy rows can still point elsewhere under storage/media/ if the database value matches.

[sjabrookes]

Thank you for the excellent explanation, @zixben!

@geourjoa: just in case it's not obvious, the Carousel images mentioned by @zixben are those that rotate on the home page, e. g. https://digipal.huma-num.fr

[geourjoa]

Thanks for the elements @zixben

My main issue is a format issue. A lot of my legacy image are in tiff format, the import script has created itemimage in JP2.
(Perhaps i have mess something, i was a bit surprised to have this issue, I will double check my legacy images folder )

---

I have write a little script to see what I have

 python3 /home/anthony/Documents/ephe/archetype/archetype3/backend/scripts/analyze_storage_extensions.py /home/anthony/Documents/ephe/archetype/archetype3/backend/storage 
Directory: /home/anthony/Documents/ephe/archetype/archetype3/backend/storage
Total files: 1257

Counts by extension:
  .tif: 999
  .jp2: 250
  .bmp: 2
  .copy: 2
  .jpg: 2
  .bk: 1
  .csv: 1

---

@pastokes show me this tool in legacy digipal, to convert image
https://github.com/kcl-ddh/digipal/blob/master/digipal/management/commands/dpim.py

Before that I have also write my own converter. It has works on few item. It's currently converting the whole stuff. My converter seem very close to the tool in legacy

[sjabrookes]

Just to confirm: back in 2011 or so, the repositories supplied most of the images in TIFF format. The first thing we did was bulk convert them to JPEG-2000. The conversion changed the colours slightly but maybe this time it’ll be better ;-)

Is there a reason to convert them rather than importing the TIFFs?

[geourjoa]

@saad-mhmd

[saad-mhmd]

TIFF → JPEG-2000 conversion for the IIIF images

I reproduced the server's broken state locally and validated a fix at full scale: all 1005 TIFFs converted, 0 failures, and every DB-referenced image renders end-to-end (994/995 rows — the one exception is a pre-existing corrupt DB path, see below).

Script: backend/scripts/convert_tif_to_jp2.py (PR #121)

How this was validated locally

Pointed my local dev stack (same compose setup) at a fresh copy of the legacy image set, reproducing the server's starting state: unconverted TIFFs, DB rows pointing to .jp2. Ran the flow below — convert → audit → tile tests → reindex. Result: 1005/1005 converted (989 vips, 15 Pillow-recovered, 1 skip), 994/995 DB rows resolve, thumbnails and deep zoom verified in the app.

The problem

Checked from outside via the public IIIF URLs: SIPI responds fine but can't serve the images — they don't appear to be on the server yet:

https://archetype-iiif.tetras-libre.fr/cccc%2F419%2F40.jp2/info.json  -> 404 "Image file not accessible"
(same for .tif, and across other collections)

The DB rows already point to .jp2, which is the right target — no DB change is needed. The task: put the source images on the server, convert, reindex.

Why convert to jp2 instead of serving the TIFFs

Nearly all legacy TIFFs are JPEG-compressed-in-TIFF (compression type 7), which SIPI v5.0.1 cannot decode. Trap: info.json returns 200 (header-only read) but tile requests fail with 500 — so always verify with a real tile request, not info.json. SIPI serves .jp2 natively; the script converts losslessly (no quality loss, no colour shift).

The script

For every .tif under the media root, writes a lossless .jp2 sibling. Converts via vips; automatically falls back to Pillow for malformed files (JPEG-in-TIFF with an alpha channel — RECOVERED (pillow) in the output is expected, not an error; 15 files in the full run). Collision-safe, non-destructive (keeps the .tif), idempotent (re-run resumes).

Steps on the server

Prerequisite: the source .tif files must be under storage/media/ at their DB-relative paths (e.g. cccc/419/40.tif) first — the script converts what's on disk, it doesn't fetch. Disk: ~60 GB free during conversion; ~32 GB after removing the TIFFs.

All commands below run on the server host, from the deploy directory (the one containing compose.yaml and storage/):

sudo apt-get install -y libvips-tools python3-pil    # or: pip install pillow

# the backend runs as a Docker image on the server, so fetch the script from the repo
# (from main once this PR is merged)
curl -LO https://raw.githubusercontent.com/archetype-pal/backend/main/scripts/convert_tif_to_jp2.py
python3 convert_tif_to_jp2.py storage/media

# audit: every DB path resolves to a file (expected: 1 miss, the corrupt row `jjp2_1`)
docker compose exec -T postgres psql -U postgres -d <dbname> -t -A \
  -c "SELECT image FROM manuscripts_itemimage WHERE image <> '';" | \
  while IFS= read -r p; do [ -f "storage/media/$p" ] || echo "MISSING: $p"; done

# smoke-test a real TILE — expect 200
curl -s -o /dev/null -w '%{http_code}\n' \
  "https://archetype-iiif.tetras-libre.fr/cccc%2F419%2F40.jp2/full/300,/0/default.jpg"

# reindex (Meilisearch bakes the IIIF URLs at index time)
just reindex

SIPI caches served images and doesn't re-check the source. If a file it already served is replaced on disk, recreate the container: docker compose up -d --force-recreate image_server.

Optional cleanup once everything renders (~24 GB back):

cd storage/media
while IFS= read -r -d '' f; do
  jp2="${f%.tif}.jp2"; [ -s "$jp2" ] && rm -f "$f"
done < <(find . -type f -name '*.tif' -print0)

Known gaps (pre-existing, out of scope)

• 1 DB row with a corrupt image path (jjp2_1) — needs a manual fix or deletion.
• ~260 image files with no DB row (whole collections). Creating those rows belongs to the metadata migration (archetype-data-migration#1); the files are self-describing — keep them on disk.