When preprocessing an observation where a coarse channel ends early, birli is not correctly averaging the final timestamp on that coarse channel.
we'll use mwa-demo to test this
git clone https://github.com/mwaTelescope/mwa-demo
cd mwa-demo
mkdir -p demo/data/1128352792/raw
curl -L -o demo/data/1128352792/raw/1128352792_20151008151937_gpubox07_00.fits 'https://projects.pawsey.org.au/birli-test/1128352792_20151008151937_gpubox07_00.fits'
curl -L -o demo/data/1128352792/raw/1128352792_20151008151937_gpubox08_00.fits 'https://projects.pawsey.org.au/birli-test/1128352792_20151008151937_gpubox08_00.fits'
birli preprocess only first antenna and last few timesteps, no averaging (2s)
mkdir -p demo/data//1128352792/prep/
birli \
-m demo/data/1128352792/raw/1128352792.metafits \
--sel-time 50 53 --sel-ants 0 --flag-edge-width 80 \
-u demo/data//1128352792/prep/birli_2s_1128352792.uvfits \
demo/data//1128352792/raw/1128352792_2*.fits
Scheduled start: 2015-10-08 15:19:35.000 UTC, unix=1444317575.000, gps=1128352792.000, mjd=4951034375.000, lmst= 3.5080°, lmst2k= 3.3100°, lat2k=-26.7907°
Scheduled end: 2015-10-08 15:21:27.000 UTC, unix=1444317687.000, gps=1128352904.000, mjd=4951034487.000, lmst= 3.9759°, lmst2k= 3.7783°, lat2k=-26.7907°
Scheduled duration: 112.000s = 56 * 2.000s
Quack duration: 2.000s = 1 * 2.000s
Output duration: 8.000s = 4 * 2.000s
Scheduled Bandwidth: 30.720MHz = 24 * 32 * 40.000kHz
Output Bandwidth: 2.560MHz = 64 * 40.000kHz
Timestep details (all=56, provided=54, common=53, good=52, select=4, flag=4):
2015-10-08 UTC + unix [s] gps [s] p c g s f
ts0: 15:19:35.000 1444317575.000 1128352792.000 p c f
ts1: 15:19:37.000 1444317577.000 1128352794.000 p c g
...
ts50: 15:21:15.000 1444317675.000 1128352892.000 p c g s
ts51: 15:21:17.000 1444317677.000 1128352894.000 p c g s
ts52: 15:21:19.000 1444317679.000 1128352896.000 p c g s
ts53: 15:21:21.000 1444317681.000 1128352898.000 p s f
Coarse channel details (metafits=24, provided=2, common=2, good=2, select=2, flag=22, ranges=1):
gpu corr rec cen [MHz] p c g s f range
...
cc16: 8 7 147 188.1600 p c g s 0 (0-23)
cc17: 7 6 148 189.4400 p c g s 0 (0-23)
... plot and export auto magnitudes to tsv
demo/04_ssins.py demo/data/1128352792/prep/birli_2s_1128352792.uvfits --no-diff --threshold 9999 --export-tsv
gps_time 187.5350 187.5750 187.6150 187.6550 187.6950 187.7350 187.7750 187.8150 187.8550 187.8950 187.9350 187.9750 188.0150 188.0550 188.0950 188.1350 188.1750 188.2150 188.2550 188.2950 188.3350 188.3750 188.4150 188.4550 188.4950 188.5350 188.5750 188.6150 188.6550 188.6950 188.7350 188.7750 188.8150 188.8550 188.8950 188.9350 188.9750 189.0150 189.0550 189.0950 189.1350 189.1750 189.2150 189.2550 189.2950 189.3350 189.3750 189.4150 189.4550 189.4950 189.5350 189.5750 189.6150 189.6550 189.6950 189.7350 189.7750 189.8150 189.8550 189.8950 189.9350 189.9750 190.0150 190.0550
1128352893.000 -0.000 -0.001 -0.002 -0.005 0.004 -0.003 0.002 -0.005 0.005 0.007 -0.003 -0.000 -0.003 -0.003 0.008 0.003 -0.001 -0.001 -0.007 -0.003 0.004 -0.008 0.003 -0.002 -0.001 -0.007 -0.004 -0.001 -0.002 0.002 -0.005 0.002 0.444 0.445 0.442 0.437 0.444 0.447 0.447 0.440 0.448 0.445 0.436 0.436 0.441 0.450 0.433 0.437 0.435 0.433 0.446 0.433 0.443 0.442 0.437 0.443 0.440 0.445 0.444 0.439 0.436 0.449 0.430 0.428
1128352895.000 -0.004 -0.001 0.002 0.008 -0.003 -0.006 -0.002 -0.000 0.002 0.002 -0.000 0.000 -0.006 0.007 -0.002 0.001 0.002 -0.007 0.003 0.006 -0.003 0.000 -0.000 -0.002 0.002 -0.000 -0.002 -0.005 0.006 0.004 0.008 0.001 0.435 0.434 0.445 0.442 0.433 0.437 0.442 0.442 0.440 0.451 0.443 0.438 0.438 0.428 0.445 0.438 0.433 0.436 0.442 0.447 0.432 0.445 0.443 0.438 0.444 0.444 0.442 0.441 0.444 0.435 0.447 0.453
1128352897.000 0.000 0.006 -0.003 -0.004 -0.002 0.002 0.000 0.004 -0.004 -0.006 0.005 0.000 0.005 -0.005 -0.008 -0.003 -0.001 -0.001 -0.003 0.002 -0.000 -0.000 -0.001 -0.000 0.005 -0.001 0.012 0.007 0.001 -0.005 0.004 -0.001 0.444 0.444 0.437 0.445 0.447 0.440 0.434 0.442 0.435 0.427 0.444 0.449 0.445 0.446 0.445 0.448 0.456 0.455 0.435 0.444 0.449 0.437 0.443 0.443 0.440 0.434 0.438 0.444 0.444 0.439 0.446 0.443
1128352899.000 0.004 -0.004 0.003 0.001 0.001 0.007 -0.000 0.001 -0.003 -0.003 -0.002 -0.000 0.005 0.001 0.002 -0.001 0.000 0.010 0.007 -0.005 -0.001 0.008 -0.002 0.004 -0.006 0.008 -0.005 -0.001 -0.005 -0.000 -0.007 -0.001 -1.324 -1.324 -1.324 -1.324 -1.324 -1.324 -1.324 -1.324 -1.324 -1.324 -1.324 -1.324 -1.324 -1.324 -1.324 -1.324 -1.324 -1.324 -1.324 -1.324 -1.324 -1.324 -1.324 -1.324 -1.324 -1.324 -1.324 -1.324 -1.324 -1.324 -1.324 -1.324
the last timestep on channel 148 is all negative, but that's fine because it's all flagged.
birli preprocess only first antenna and last few timesteps, averaging to 4s
birli \
-m demo/data/1128352792/raw/1128352792.metafits \
--sel-time 50 53 --sel-ants 0 --flag-edge-width 80 \
--avg-time-res 4 \
-u demo/data//1128352792/prep/birli_4s_1128352792.uvfits \
demo/data//1128352792/raw/1128352792_2*.fits
demo/04_ssins.py demo/data/1128352792/prep/birli_4s_1128352792.uvfits --no-diff --threshold 9999 --export-tsv
gps_time 187.5350 187.5750 187.6150 187.6550 187.6950 187.7350 187.7750 187.8150 187.8550 187.8950 187.9350 187.9750 188.0150 188.0550 188.0950 188.1350 188.1750 188.2150 188.2550 188.2950 188.3350 188.3750 188.4150 188.4550 188.4950 188.5350 188.5750 188.6150 188.6550 188.6950 188.7350 188.7750 188.8150 188.8550 188.8950 188.9350 188.9750 189.0150 189.0550 189.0950 189.1350 189.1750 189.2150 189.2550 189.2950 189.3350 189.3750 189.4150 189.4550 189.4950 189.5350 189.5750 189.6150 189.6550 189.6950 189.7350 189.7750 189.8150 189.8550 189.8950 189.9350 189.9750 190.0150 190.0550
1128352894.000 -0.002 -0.001 0.002 0.002 0.001 -0.003 -0.000 -0.003 0.004 0.005 -0.003 -0.000 -0.005 0.004 0.006 0.002 0.000 -0.001 0.000 -0.001 0.000 -0.004 0.001 -0.001 -0.002 -0.002 -0.007 -0.005 0.000 0.004 0.002 0.001 0.440 0.440 0.003 -0.002 -0.003 0.001 0.004 -0.000 0.444 0.008 -0.002 -0.004 0.439 0.439 -0.002 -0.004 0.434 -0.008 0.003 0.440 0.437 0.443 0.440 0.441 0.442 0.445 0.002 0.440 0.440 0.001 0.439 0.441
1128352898.000 0.002 0.001 -0.002 -0.002 -0.001 0.003 0.000 0.003 -0.004 -0.005 0.003 0.000 0.005 -0.004 -0.006 -0.002 -0.000 0.001 -0.000 0.001 -0.000 0.004 -0.001 0.001 0.002 0.002 0.007 0.005 -0.000 -0.004 -0.002 -0.001 -0.440 -0.440 -0.003 0.002 0.003 -0.001 -0.004 0.000 -0.444 -0.008 0.002 0.004 -0.439 -0.439 0.002 0.004 -0.434 0.008 -0.003 -0.440 -0.437 -0.443 -0.440 -0.441 -0.442 -0.445 -0.002 -0.440 -0.440 -0.001 -0.439 -0.441
Now some negative values (not all) have made it into the last timestamp, and because not all of the visibilities that went into these were all flagged, these values are not flagged, they're just down-weighted.
Average should not be including these flagged values.
When preprocessing an observation where a coarse channel ends early, birli is not correctly averaging the final timestamp on that coarse channel.
we'll use mwa-demo to test this
birli preprocess only first antenna and last few timesteps, no averaging (2s)
mkdir -p demo/data//1128352792/prep/ birli \ -m demo/data/1128352792/raw/1128352792.metafits \ --sel-time 50 53 --sel-ants 0 --flag-edge-width 80 \ -u demo/data//1128352792/prep/birli_2s_1128352792.uvfits \ demo/data//1128352792/raw/1128352792_2*.fitsScheduled start: 2015-10-08 15:19:35.000 UTC, unix=1444317575.000, gps=1128352792.000, mjd=4951034375.000, lmst= 3.5080°, lmst2k= 3.3100°, lat2k=-26.7907° Scheduled end: 2015-10-08 15:21:27.000 UTC, unix=1444317687.000, gps=1128352904.000, mjd=4951034487.000, lmst= 3.9759°, lmst2k= 3.7783°, lat2k=-26.7907° Scheduled duration: 112.000s = 56 * 2.000s Quack duration: 2.000s = 1 * 2.000s Output duration: 8.000s = 4 * 2.000s Scheduled Bandwidth: 30.720MHz = 24 * 32 * 40.000kHz Output Bandwidth: 2.560MHz = 64 * 40.000kHz Timestep details (all=56, provided=54, common=53, good=52, select=4, flag=4): 2015-10-08 UTC + unix [s] gps [s] p c g s f ts0: 15:19:35.000 1444317575.000 1128352792.000 p c f ts1: 15:19:37.000 1444317577.000 1128352794.000 p c g ... ts50: 15:21:15.000 1444317675.000 1128352892.000 p c g s ts51: 15:21:17.000 1444317677.000 1128352894.000 p c g s ts52: 15:21:19.000 1444317679.000 1128352896.000 p c g s ts53: 15:21:21.000 1444317681.000 1128352898.000 p s f Coarse channel details (metafits=24, provided=2, common=2, good=2, select=2, flag=22, ranges=1): gpu corr rec cen [MHz] p c g s f range ... cc16: 8 7 147 188.1600 p c g s 0 (0-23) cc17: 7 6 148 189.4400 p c g s 0 (0-23) ...plot and export auto magnitudes to tsv
the last timestep on channel 148 is all negative, but that's fine because it's all flagged.
birli preprocess only first antenna and last few timesteps, averaging to 4s
birli \ -m demo/data/1128352792/raw/1128352792.metafits \ --sel-time 50 53 --sel-ants 0 --flag-edge-width 80 \ --avg-time-res 4 \ -u demo/data//1128352792/prep/birli_4s_1128352792.uvfits \ demo/data//1128352792/raw/1128352792_2*.fits demo/04_ssins.py demo/data/1128352792/prep/birli_4s_1128352792.uvfits --no-diff --threshold 9999 --export-tsvNow some negative values (not all) have made it into the last timestamp, and because not all of the visibilities that went into these were all flagged, these values are not flagged, they're just down-weighted.
Average should not be including these flagged values.