Repository navigation
docs(design): plan the FITS profile, a single-file virtualizer with a tile codec - #88
Merged
Merged
Conversation
… tile codec Add design/plans/fits.md: the research on how astronomy archives use FITS (one self-contained file per deliverable; multi-extension files as the norm; tile compression in the ground-based archives; siblings related by names, not by the file), the decision to take a single file and leave collections to a later combine step, and the implementation plan: the convention (HDU nodes by index, images as big-endian contiguous values, tables as bytes, typed cards with a budget), tile-compressed images as chunks of a literal per-tile prefix plus one heap range decoded by a new `fits_tiles` codec, the source model, the profile's checks and reading rounds, the Rust core modules, tests, fixtures, the astropy verifier, a public corpus, the order of work, and twelve decisions with recommended answers. Assisted-by: ClaudeCode:claude-fable-5-1 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…he FITS profile Add the FITS profile (spec/virtualize/fits.md, §15 of spec/virtualize.md) for uncompressed files: a FITS file is chosen by `SIMPLE =` and a fixed-format `T` in column 30, its HDUs are walked by their headers alone, and each becomes a node at the path of its index: an array for an image (big-endian `bytes`, contiguous values cut with a 1 MiB limit, the stored values with BSCALE/BZERO/BLANK recorded), a group with `rows` and `heap` byte arrays for a table, and an empty group for an HDU without data. The root lists the HDUs with their offsets, and every node carries its header's cards as typed [keyword, value, comment] triples under a 64 KB budget; refkv_source is the file's IR mirror (one struct per HDU: header, data or rows/gap/heap, padding), from which the file is rebuilt byte for byte. Tile-compressed images, random groups, non-conforming files and gzip or bzip2 files are rejected. The Rust core gets the parser (src/fits.rs: cards with CONTINUE and HIERARCH, the HDU walk over a sliding 64 KiB window), its source model (schema/fits.json) and the projection (project/fits.rs); the format reaches Python, the browser and the CLI by name through the registry. astropy writes the synthetic fixtures (fixtures/fits: 31 accepted, 27 rejected) and is the independent reader of python/tests/ir/test_fits.py; tests/parsers.rs checks each rejection's message. The design is in design/plans/fits.md. Assisted-by: ClaudeCode:claude-fable-5-1 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…e does An HDU is empty only when NAXIS = 0; an image with a zero-length axis is an array with no chunks, as the parser and the convention now say. Assisted-by: ClaudeCode:claude-fable-5-1 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…44b789 The VSI profile took section 15 on main; FITS is section 16. Assisted-by: ClaudeCode:claude-fable-5-1 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
… decoded by the Rust core A BINTABLE with ZIMAGE = T (what fpack writes to .fz files) becomes an array of the image's shape and type, chunked by runs of tiles up to 1 MiB decoded, cut as conventions §7 cuts contiguous values over the tile grid. Each chunk is a literal prefix (every tile's stream length and kind, ZBLANK, ZSCALE and ZZERO) followed by one byte range of the heap, since fpack and astropy store tiles one after the other (per-tile ranges when they do not). The parser reads the main table in one range, checks the ZCMPTYPE, ZBITPIX, ZNAXISn, ZTILEn, RICE, ZQUANTIZ, ZDITHER0, column, row-count, descriptor and tile-order rules, and describes each chunk's heap bytes as an element. The `fits_tiles` codec (rust/refkv-ir/src/fits_tiles.rs) decodes RICE_1 (a port of cfitsio's fits_rdecomp, _short and _byte), GZIP_1, GZIP_2 and NOCOMPRESS tiles, gzip-fallback tiles, and dequantizes with cfitsio's random table and dither (unquantize.c), bit for bit against astropy's codec; it is exposed to Python (_core.fits_tiles_decode, refkv.zarr.codecs.FitsTilesCodec) and to wasm (vz_fits_tiles_decode). The convention computes the dequantization unfused in binary64; astropy's arm64 build fuses the multiply and add, so the tests compare quantized floats within that rounding. Also: the FITS spec gains §3.3 and the compressed checks of §16.3, the schema the root table's compression member; 28 compressed fixtures (astropy-written and hand-built: gzip fallback, absent tile, heap gaps, Q descriptors, BLOCKSIZE 16, no ZTILE cards) and 17 rejections; js/test/fits/verify.py (the browser twin against astropy); docs/virtualize/fits/README.md, the demo's FITS examples and conformance/virtualize/corpus_fits.txt. FITS is section 16, after main's VSI profile. Assisted-by: ClaudeCode:claude-fable-5-1 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
… harness keys them by one suffix The corpus comparison keys fixtures by their last suffix; the registry's suffixes are .fits and .fz, and the two compressed-file fixtures are rejected by their bytes. Assisted-by: ClaudeCode:claude-fable-5-1 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…44b789 The QPTIFF profile took section 16 on main; FITS is section 17. Assisted-by: ClaudeCode:claude-fable-5-1 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…endent code The fixtures-reproducible check differed on Linux: gzip members from Python's zlib and astropy's float quantization (whose noise estimate depends on the build's arithmetic) are not the same bytes on every platform. The generator now writes gzip members as stored deflate blocks and quantizes by hand (random 32-bit integers per tile with per-tile ZSCALE and ZZERO, a ZBLANK card or column, dither 2's zero value next to integers near -2^31); astropy remains the decoding oracle. One more fixture, fits_comp_zblank_column. Assisted-by: ClaudeCode:claude-fable-5-1 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…44b789 The MP3 profile took section 17 on main; FITS is section 18. The codec is renamed refkv.fits_tiles, as refkv's own codecs carry the refkv. prefix. Assisted-by: ClaudeCode:claude-fable-5-1 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This was referenced Oct 24, 2026
Closed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
🤖 AI text below 🤖
Adds
design/plans/fits.md, the research and implementation plan for a FITS profile. The agent surveyed how astronomy archives use FITS (SDSS, HST, JWST, Rubin, Euclid, DECam, Pan-STARRS, Legacy Surveys, ZTF, TESS, DESI, HiPS, radio), inspected real files by hand, and concluded that the profile should take a single file, with collections left to a later combine step outside any profile, because every archive's deliverable is a self-contained file, there is no manifest relating siblings, and a multi-extension file already is a collection.What the plan specifies:
iis the nodei; images are big-endian contiguous values cut into 1 MiB chunks; tables are kept asrowsandheapbyte arrays with their column definitions; cards are typed triples on each HDU's node under a 64 KB budget, with the raw header in the mirror..fz): chunks are a literal prefix of per-tile records (length, kind, ZBLANK, ZSCALE, ZZERO, and the first tile's storage index) followed by one heap byte range, decoded by a newfits_tilescodec. The prefix carries everything the dither needs, so the decoder never needs the chunk's position (the point raised in Varying codec parameters per-chunk zarr-developers/zarr-specs#399). The grid is cut by decoded size, so a.fzfile and its unpacked twin share a chunk grid. Verified on a DECam exposure: 249,734 tiles stored consecutively with no gaps, and every tile decodable from its own range.Reviewer notes: section 11 holds the decisions worth a look (unsigned integers recorded rather than presented as
uint16; tables as bytes in version 0; the xarray dataset layout deferred; HCOMPRESS, PLIO, random groups and whole-file gzip or bzip2 rejected). Two items are marked to confirm during implementation: whether thericecompcrate (MIT) builds for wasm, and whether the S3 and NOIRLab corpus URLs serve anonymous ranges with CORS.🤖 Generated with Claude Code