Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 11 additions & 0 deletions .dockerignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
.coverage/
.dist/
.git/
.github/
.idea/
.pytest_cache/
.ruff_cache/
.venv/
.vscode/
__pycache_/
target/
57 changes: 57 additions & 0 deletions .github/workflows/docker.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,57 @@
# .github/workflows/docker-publish.yml
name: Build and publish container

on:
push:
branches: [main]
tags: ['v*.*.*']
pull_request:
branches: [main]
workflow_dispatch:

env:
IMAGE_NAME: mwatelescope/mwalib

permissions:
contents: read

jobs:
docker:
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: actions/checkout@v6

- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v4

- name: Log in to Docker Hub
if: github.event_name != 'pull_request'
uses: docker/login-action@v4
with:
username: ${{ secrets.DOCKERHUB_USERNAME }}
password: ${{ secrets.DOCKERHUB_TOKEN }}

- name: Extract image metadata (tags, labels)
id: meta
uses: docker/metadata-action@v6
with:
images: ${{ env.IMAGE_NAME }}
tags: |
type=semver,pattern={{version}}
type=semver,pattern={{major}}.{{minor}}
type=raw,value=latest,enable={{is_default_branch}}
type=sha,format=short

- name: Build and push
uses: docker/build-push-action@v7
with:
context: .
push: ${{ github.event_name != 'pull_request' }}
tags: ${{ steps.meta.outputs.tags }}
labels: ${{ steps.meta.outputs.labels }}
annotations: ${{ steps.meta.outputs.annotations }}
provenance: true
sbom: true
cache-from: type=gha
cache-to: type=gha,mode=max
52 changes: 26 additions & 26 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -1,30 +1,30 @@
.coverage
.python-version
__pycache__/
.idea/
.vscode/
Cargo.lock
/.idea/
target
/examples/build/
/examples/sum-all-hdus
/examples/mwalib-print-context
/examples/mwalib-sum-all-hdus
/examples/mwalib-print-volt-context
/examples/mwalib-sum-vcs
.vscode
/coverage
/tools/release
target/
examples/build/
examples/sum-all-hdus
examples/mwalib-print-context
examples/mwalib-sum-all-hdus
examples/mwalib-print-volt-context
examples/mwalib-sum-vcs
coverage/
tools/release/
include/
/tools/comparison_tools/*.csv
/tools/comparison_tools/.python-version
/test_files/1101503312_1_timestep/*.sub
/test_files/1101503312_1_timestep/*.dat
tools/comparison_tools/*.csv
tools/comparison_tools/.python-version
test_files/1101503312_1_timestep/*.sub
test_files/1101503312_1_timestep/*.dat
ccov.zip
/test_files/1101503312_mwax_vcs/*.sub
/test_files/1101503312_vcs/*.dat
/test_files/1370755832_mwax_vcs_os/*.sub
/test_files/1380365160/*.sub
test_files/1101503312_mwax_vcs/*.sub
test_files/1101503312_vcs/*.dat
test_files/1370755832_mwax_vcs_os/*.sub
test_files/1380365160/*.sub
test_files/1363085416/*.sub
.coverage
.python-version
__pycache__
/tmp
/examples/*.o
/examples/*.dbg
/dist
tmp/
examples/*.o
examples/*.dbg
dist/
17 changes: 17 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,23 @@ Notes:
* Changes tagged with "FFI/C" are only relevant if you are using mwalib's C library (you are developing in C/C++).
* Changed taged with "Python" are only relevant if you are using mwalib via Python.

## 3.0.0 04-Sep-2026

### Breaking

* Bumped MSRV to 1.88.0
* `convert_gpstime_to_unixtime` and `convert_unixtime_to_gpstime` (in `misc`, and re-exported at the crate root) no longer take `mwa_start_gpstime_ms`/`mwa_start_unixtime_ms` reference parameters - they are now `convert_gpstime_to_unixtime(gpstime_ms: u64)` and `convert_unixtime_to_gpstime(unixtime_ms: u64)`. Internally they now convert via `hifitime`'s leap-second table instead of a flat per-observation offset, which also removes the previous assumption that an observation could never span a leap second boundary.
* `chrono` has been removed entirely. `MetafitsContext::sched_start_utc`/`sched_end_utc` and `VoltageBeam::modtime` are now `jiff::Timestamp` instead of `chrono::DateTime<FixedOffset>`. `jiff::Timestamp`.
* `MetafitsContext`'s `Display` output now prints `Scheduled start (utc)` and `Scheduled end (utc)` using `jiff::Timestamp`'s own `Display`, which is genuine RFC3339 (e.g. `2014-12-01T21:08:16Z`) - unlike `chrono`'s default `Display` format (`2014-12-01 21:08:16 +00:00`, space-separated and not itself RFC3339-compliant). This is a deliberate move to a standards-compliant format.
* Python: these fields continue to surface as native `datetime.datetime` objects with no change to the Python-visible type or semantics. `jiff::Timestamp` already has working `IntoPyObject`/`FromPyObject` via `pyo3`'s own `jiff-02` feature (the same mechanism `chrono` previously used).
* FFI/C: no change - these fields were already exposed as `time_t`, computed the same way (seconds since the UNIX epoch, now via `Timestamp::as_second()` instead of `chrono`'s `.timestamp()`).

### Changed

* Added `hifitime` as a dependency to back the above leap-second-aware conversions.
* Added `jiff` as a dependency so the field types on `MetafitsContext`/`VoltageBeam` are consistent regardless of which features are enabled to back the `chrono` removal above. `hifitime` and `jiff` serve different purposes in mwalib: `hifitime` for leap-second/GPS-time-scale-aware arithmetic, `jiff` for the plain UTC timestamp fields that need to interoperate cleanly with Python's `datetime.datetime`.
* Updated Dockerfile and added Docker build CI

## 2.1.0 01-Sep-2026

### Added
Expand Down
23 changes: 14 additions & 9 deletions Cargo.toml
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
[package]
name = "mwalib"
version = "2.1.0"
version = "3.0.0"
homepage = "https://github.com/MWATelescope/mwalib"
repository = "https://github.com/MWATelescope/mwalib"
readme = "README.md"
Expand All @@ -9,7 +9,7 @@ authors = [
"Christopher H. Jordan <christopherjordan87@gmail.com>",
]
edition = "2021"
rust-version = "1.85.0"
rust-version = "1.88.0"
resolver = "3"
description = "A library to simplify reading Murchison Widefield Array (MWA) raw visibilities, voltages and metadata."
license = "MPL-2.0"
Expand Down Expand Up @@ -57,9 +57,10 @@ opt-level = 3

[dependencies]
bytemuck = "1.24"
chrono = "0.4"
fitsio = "0.21"
fitsio-sys = "0.5"
hifitime = "4.3"
jiff = "0.2"
libc = "0.2"
log = "0.4"
num-derive = "0.5"
Expand All @@ -73,23 +74,23 @@ env_logger = { version = "0.11", optional = true }

# "python" feature
ndarray = { version = "0.17", optional = true }
numpy = { version = "0.28", optional = true }
pyo3 = { version = "0.28", features = [
"chrono",
numpy = { version = "0.29", optional = true }
pyo3 = { version = "0.29", features = [
"extension-module",
"jiff-02",
"macros",
], optional = true }

# "python-stubgen" feature
pyo3-stub-gen = { version = "0.22", optional = true }
pyo3-stub-gen-derive = { version = "0.22", optional = true }
pyo3-stub-gen = { version = "=0.23.1", optional = true, features = ["jiff-02"] }
pyo3-stub-gen-derive = { version = "0.23", optional = true }

# Transitive due to latest versions using higher MSRV than mwalib
# "examples" feature.
clap = { version = "4.6", features = ["derive"], optional = true }

[dev-dependencies]
criterion = "0.7"
criterion = "0.8"
csv = "1.4"
float-cmp = "0.10"
tempfile = "3.24"
Expand Down Expand Up @@ -125,3 +126,7 @@ required-features = ["examples"]
[[example]]
name = "mwalib-sum-first-fine-channel-gpubox-hdus"
required-features = ["examples"]

[patch.crates-io]
# Use a specific commit of pyo3-stub-gen to enable compatibibility with jiff (remove this patch once this PR https://github.com/Jij-Inc/pyo3-stub-gen/pull/496 is merged)
pyo3-stub-gen = { git = "https://github.com/cjdsellers/pyo3-stub-gen", rev = "1b4f1cc566cb58e16e19c79045d67f345e9ef8e6" }
131 changes: 90 additions & 41 deletions Dockerfile
Original file line number Diff line number Diff line change
@@ -1,50 +1,58 @@
FROM python:3.12-slim-bookworm AS base
# syntax=docker/dockerfile:1

# --------------------------------------------------------------------------
# Multi-stage build:
# builder - has the full toolchain (rustc/cargo, cmake, build-essential,
# cfitsio -dev headers) needed to compile mwalib, run its tests,
# and build the python wheel + venv.
# runtime - a clean python:3.13-slim-trixie image that copies over only
# the finished artifacts (venv, compiled cfitsio .so, example
# binaries) - no compilers, headers, or Rust toolchain.
# Result: a much smaller, lower-attack-surface image; the builder stage is
# discarded entirely except for what's explicitly copied with --from=builder.
# --------------------------------------------------------------------------

# ---------- builder ----------
FROM python:3.13-slim-trixie AS builder
# Compiles everything (rust, cfitsio, python wheel) - discarded except for
# what the runtime stage below copies out with --from=builder.

# suppress perl locale errors
ENV LC_ALL=C
# suppress apt-get prompts
ENV DEBIAN_FRONTEND=noninteractive
# suppress apt-get prompts (build-time only, not left set in the final image)
ARG DEBIAN_FRONTEND=noninteractive
RUN apt-get update \
&& apt-get install -y --no-install-recommends \
&& apt-get install -y --no-install-recommends \
build-essential \
cmake \
pkg-config \
curl \
libcurl4-openssl-dev \
libz-dev \
zlib1g-dev \
&& apt-get autoclean \
&& apt-get clean \
&& apt-get autoremove -y \
&& rm -rf /var/lib/apt/lists/*

# # # install python deps for mwalib python
RUN python -m pip install --force-reinstall --no-cache-dir \
maturin[patchelf] \
pip \
numpy \
pytest

# # install cfitsio

# install cfitsio into /usr/local
ARG CFITSIO_VERSION=4.6.3
RUN cd / && \
curl "https://heasarc.gsfc.nasa.gov/FTP/software/fitsio/c/cfitsio-${CFITSIO_VERSION}.tar.gz" -o cfitsio.tar.gz && \
tar -xf cfitsio.tar.gz && \
rm cfitsio.tar.gz && \
cd cfitsio-${CFITSIO_VERSION} && \
RUN curl -fsSL "https://heasarc.gsfc.nasa.gov/FTP/software/fitsio/c/cfitsio-${CFITSIO_VERSION}.tar.gz" -o /tmp/cfitsio.tar.gz && \
tar -xf /tmp/cfitsio.tar.gz -C /tmp && \
rm /tmp/cfitsio.tar.gz && \
cd /tmp/cfitsio-${CFITSIO_VERSION} && \
cmake -S . -B build \
-DCMAKE_INSTALL_PREFIX=/usr/local \
-DCMAKE_INSTALL_PREFIX=/usr/local \
-DUSE_PTHREADS=ON \
-DUSE_SSE2=OFF \
-DUSE_SSSE3=OFF \
-DUSE_CURL=ON && \
cmake --build build -j && \
cmake --install build && \
ldconfig && \
cd / && \
rm -rf /cfitsio-${CFITSIO_VERSION}
rm -rf /tmp/cfitsio-${CFITSIO_VERSION}

# # Get Rust
ARG RUST_VERSION=1.90
# Get Rust
ARG RUST_VERSION=stable
ENV RUSTUP_HOME=/opt/rust CARGO_HOME=/opt/cargo
ENV PATH="${CARGO_HOME}/bin:${PATH}"
RUN mkdir -m755 $RUSTUP_HOME $CARGO_HOME && ( \
Expand All @@ -54,27 +62,68 @@ RUN mkdir -m755 $RUSTUP_HOME $CARGO_HOME && ( \
--default-toolchain=${RUST_VERSION} \
)

# mount cwd to /mwalib in Docker
ADD . /mwalib
# isolated venv for the python side, so it's a clean, self-contained thing
# to copy into the runtime stage
RUN python -m venv /opt/venv
ENV PATH="/opt/venv/bin:${PATH}"
RUN pip install --no-cache-dir --upgrade pip

# copy source into /mwalib
COPY . /mwalib
WORKDIR /mwalib

# Update cargo registry
# NB: this intentionally ignores Cargo.lock pins in favour of the newest
# compatible deps. Drop this RUN if you want reproducible builds instead.
RUN cargo update --verbose

# build python module and examples
RUN cargo build --examples --features=examples && \
cargo test --examples --features=examples && \
maturin build --features=python && \
python -m pip install $(ls -1 target/wheels/*.whl | tail -n 1) && \
rm -rf ${CARGO_HOME}/registry
ENV PATH=${PATH}:/mwalib/target/debug/examples/
# build + test the rust examples
RUN cargo build --features=examples && \
cargo test --features=examples

# Build and install the python module + its "dev" dependency group straight
# from pyproject.toml:
# - maturin's version comes from [build-system].requires (maturin>=1.0,<2.0)
# - the "python"/"pyo3-extension-module" features come from [tool.maturin]
# - numpy comes along automatically as a declared [project] dependency
# - pytest/ruff/toml/ty come from the "dev" group under [dependency-groups]
# (PEP 735 - needs pip>=25.1, hence the upgrade above)
RUN pip install --no-cache-dir . --group dev

# Run Python tests
RUN pytest

RUN <<EOF
#!/usr/bin/env python
import sys
from sys import implementation, stdout
print( f"{implementation=}", file=stdout)
EOF

# ---------- runtime ----------
FROM python:3.13-slim-trixie AS runtime
# Clean image: only runtime shared libs + the built venv/binaries from
# `builder` - no compilers or dev headers ship in the final image.

ARG DEBIAN_FRONTEND=noninteractive
RUN apt-get update \
&& apt-get install -y --no-install-recommends \
libcurl4 \
zlib1g \
&& apt-get autoclean \
&& apt-get clean \
&& apt-get autoremove -y \
&& rm -rf /var/lib/apt/lists/*

# cfitsio's runtime shared library, built in the builder stage (root, so it
# can update the system linker cache)
COPY --from=builder /usr/local/lib/libcfitsio* /usr/local/lib/
RUN ldconfig

# optional: run as a non-root user now that build tooling is gone.
# Drop these two lines (and the --chown flags below) to keep running as root.
RUN useradd --create-home --shell /bin/bash mwalib
USER mwalib

# the venv with mwalib + numpy already installed
COPY --from=builder --chown=mwalib:mwalib /opt/venv /opt/venv
ENV PATH="/opt/venv/bin:${PATH}"

# source tree + compiled rust example binaries
WORKDIR /mwalib
COPY --chown=mwalib:mwalib . /mwalib
COPY --from=builder --chown=mwalib:mwalib /mwalib/target/debug/examples /mwalib/target/debug/examples

RUN python -c "import sys; print(f'{sys.implementation=}')"
Loading
Loading