AOCal_viewer

Graphical viewer for AOCal and Hyperdrive format calibration files, using Napari

This command line utility allows the user to view the contents of calibration fit files, used by the Murchison Widefield Array pipelines to store calibration data for the telescope. The bulk of the work is done by napari, an open-source Python viewer for N-dimensional arrays:

https://napari.org/stable/

This utility is a wrapper that loads a Hyperdrive calibration fit file in FITS format, or an AOCal format calibration file, splits the contents into phases and gains for each polarisation product, and passes them to napari as layers, along with metadata about the observation in additional layers.

The Hyperdrive file format is described here:

https://mwatelescope.github.io/mwa_hyperdrive/defs/cal_sols_hyp.html

The AOCal file format is described here:

https://mwatelescope.atlassian.net/wiki/spaces/MP/pages/1190658052/aocal+File+Format

These files are generated by processing an MWA observation of a bright source using hyperdrive, or by using the calibration web service (described here: https://mwatelescope.atlassian.net/wiki/spaces/MP/pages/24969461/Calibration+web+services) to request one. The web service generates the AOCal file from a compressed representation of the calibration fit stored in an SQL database - which in turn, was generated from a solution generated by hyperdrive.

For both formats, the actual calibration data is an array of complex doubles (2x64-bit) of shape (num_timesteps, num_tiles, num_finechannels, 4).

Here:

• num_timesteps is always 1 for MWA calibration files.
• num_tiles depends on the telescope configuration for that observation, typically 128 or 256.
• num_finechannels is the total number of fine channels in the observation, typically 768 (24 coarse channels of 1.28 MHz each split into 32 channels of width 40 kHz).
• The final dimension is made up of X, XY, YX and YY values, in that order. Calibration files generated by the calibration web services write zeros for the cross-polarisation (XY, YX) values, while hyperdrive generated files are fully populated.

Each complex value represents a phase correction (the 'angle' of the complex value in polar representation), and a gain correction (the absolute value of the complex number, which is the vector length in polar form). On loading, calnap will convert the complex values to phases (from -Pi to +Pi) and gains (non-negative floats).

When the image processing pipeline uses an AOCal format calibration file, complex signal values are multiplied by the complex values in the calibration file, so larger gain values in the calibration file result in larger powers in the output. For those viewing the gains in the compressed database fit, these are the reciprocal (1/X) of the gain values stored in the database.

Installation

After cloning the repository:

• Create a new Python environment
• Change to the AOCal_viewer directory (where pyproject.toml is found)
• Run "pip install ."

This will place 'calnap' in your path, while the python environment is active.

Or for those using 'uv', instructions from Greg Sleap:

• Clone the repo
• Change to the AOCal_viewer directory (where pyproject.toml is found)
• run "uv sync" to create a new virtual env in .venv with all the packages
• "source .venv/bin/activate"

Or, thanks to magic comments in the file header by Dev Null, you can also cd into AOCal_viewer/src/calnap and just do:

uv run calnap.py <arguments>

And that will create and cache a custom Python environment with all the packages needed, and run the script.

Usage

The basic usage is:

calnap view <filename>

This will open up a napari window with a grey grid - the columns are the 24 coarse channels in the observation, and the rows show all the receivers in the observation, each connected to eight tiles. Frequency is on the horizontal axis (one pixel per fine channel), and input (MWA tile) is on the vertical axis (one pixel per tile). Inputs are ordered from the lowest tile ID at the top to the highest tile ID at the bottom. This is the same order as used in the data files for the observation - but bear in mind that while a calibrator observation usually uses all tiles, most modern MWA observations use some subset of tiles.

The layer tool at the bottom left of the napari window can be used to view different aspects of the data. Each line in the layer tool has an 'eye' icon to toggle the layer visibilty. Click the eye icon for a layer to make it visible, and it will brighten and show a light-gray background, indicating that this layer is visible. Click on the eye icon again to make it invisible. More than one layer can be visible at the same time.

Clicking elsewhere on a layer in the layer tool will make that layer the 'active' layer, shown by a blue background. The active layer/s can be controlled using the widgets at the top left (contrast limits, gamma factor, opacity, etc). The value of the raw data in the active layer at the pixel under the mouse cursor will be shown at the bottom left of the napari window in the data tooltip, after the coordinates. Shift-clicking will let you make multiple layers active, so widgets (eg opacity) work on all active layers, and the tooltip shows the data values in all active layers.

Note that which layer/s is/are Active (blue), and which layer/s is/are visible (highlighted eye icon) are unrelated concepts. A layer can be active but not visible, or visible and not active.

For more information on using the napari viewer, see the documentation here: https://napari.org/stable/tutorials/start_index.html

The layers are (in order):

• Channels - these are the tall, narrow vertical grey shapes showing each of the 24 coarse channels. They are passed to Napari as shapes (corner coordinates), so they have no corresponding data to show in the tooltip.
• Receivers - these are the wide grey shapes extending across the width of the plot, each having 8 rows of pixels corresponding to the 8 tiles on that receiver. This is also a 'shape' layer, with no tooltip data. The borders of the 'Channels' and 'Receivers' shapes can obscure the underlying data, so you may want to make one or both of these layer invisible.
• Tile ID: This is a label layer, so has nothing to show (the 'visible' state is ignored), but when active, the ID of the tile (eg 11-18, 21-28, ... 2001-2008) is shown in the tooltip for the pixel under the mouse pointer.
• Receiver ID: Like the Tile ID layer, only showing the receiver number (1-34) that the tile under the cursor is connected to.
• Phase:Y, Phase:YX, Phase:XY and Phase:X: These are the phase components for each polarisation term. They are represented using the 'hsv' cyclic colour map, ranging over red-orange-yellow-green-cyan for values from -Pi to zero, then cyan-blue-purple-magenta-red for values from zero to Pi. If the phase correction for a tile is small (a small residual in the effective cable length to that tile), you will see very slow phase changes across the full range of fine channels. A large residual in the effective cable length will show up as faster phase wraps over the frequency range.
• Gains:Y, Gains:YX, Gains:XY and Gains:X: These are the gain components for each polarisation term. Larger values here result in larger powers in the output data. Values near zero mean that output has effectively no signal - cross-polarisation gains (XY,YX) tend to have much lower gains than X and Y.

Command line options

The view command accepts the following options:

• --channel=N - used to indicate that the AOCal file only contains data for a single coarse channel, and to specify which coarse channel it is. Hyperdrive writes single-channel files for each coarse channel for use inside the MWAX correlator, and where the observation has 24 non-contiguous channels instead of a block of of 24 contiguous channels. Alternatively, if used on an AOCal file containing all 24 coarse channels, only coarse channel specified will be shown.
• --divide_index=N - if supplied, phases in the AOCal file will be displayed relative to the phase of the tile specified here, at the same fine channel. Note that the index is 0-num_tiles, not the Tile ID value. You need to use this option to compare Hyperdrive generated AOCal files (which have an absolute phase shift for each tile/channel) with ones generated by the web service, which contain zero phase shift for the reference tile (over all channels).
• --show_tiledata - if supplied, calnap will dump one line of text per tile when started, showing sample phase values and estimated phase slopes. You would specify this when opening a web-service generated AOCAL file, then look for the tile index with zero phase. Then specify that value as the --divide_index when opening a hyperdrive-generated AOCal file for the same observation.

Difference view comparing two calibration fit files

As well as viewing a single calibration fit file, you can also display the difference between two calibration fit files using the 'diff' command.

The phase angles shown are the phases in the second file subtracted from the phases in the first file, folded to the range -Pi to Pi. Zero means the phases are identical, and non-zero values are the difference in radians between corresponding values.

The gain values shown are the log to base 10 of the gain ratio - the gain values in the first file divided by the gain values in the second file. Again, a value of zero means there was no difference (Log10(1.0) = 0.0), but negative values mean the gains in the first file are smaller, and positive values mean the gains in the first file are larger. If a value is 1.0, for example, then the gain in the first file is ten times larger than the corresponding value in the second file.

Usage

To compare two calibration fit files, run:

calnap diff <filename1> <filename2>

The layers are the same as in the 'view' command, but with labels indicating that they show phase differences, and log10(G1/G2) gain ratios.

Command line options

The diff command accepts the following options:

• --channel=N - used to indicate that one or both AOCal files only contains data for a single coarse channel, and to specify which coarse channel it is. If one or both AOCal files contain all 24 coarse channels, only the channel specified here will be shown.
• --divide_index1=N and --divide_index2=N refer to the first and second filename on the command line respectively, and as above, are used to show hyperdrive-generated files but display phases relative to a single reference tile.