Integrate audit

[iliasoroka1]

Feature and Algorithm Work Before the LLM-Assisted Audit

At the macro level I added experimental pipeline:

define plate
    → estimate a useful scan Z
    → scan the circular plate
    → detect and center a worm
    → switch camera settings
    → refine focus
    → start tracking
    → optionally record

Automated Plate Scanning

Plate geometry

To fit a plate I fit a circle using:

$$ x_i^2 + y_i^2 + D x_i + E y_i + F = 0. $$

And estimate the coefficients with least squares regression. The centre and radius are

$$ c_x=-\frac{D}{2}, \qquad c_y=-\frac{E}{2}, \qquad r=\sqrt{c_x^2+c_y^2-F}. $$

tile generation

I use calibrated camera field of view to calculate distances between scan
positions.

$$ \Delta x = \max(W(1-o_x), 10^{-3}), \qquad \Delta y = \max(H(1-o_y), 10^{-3}). $$

Then I reaine a tile only if its centre satisfies circle

$$ (x-c_x)^2 + (y-c_y)^2 \leq r^2. $$

While scanning, rows alternate between left to right and right to left

Scan-to-track transition

Once a worm is centred I do following

• switch to the tracking/recording exposure, gain, and frame rate
• starts live view
• starts tracking and live autofocus
• starts recording

I repeat the scan if a worm is not found

Autofocus and Focus-Plane Estimation

I changed the autofocus to be a peek seekeer rather then a PID

At each Z position it collects (B) frame-level focus scores:

$$ f_1,f_2,\ldots,f_B. $$

And calculates the median for that Z:

$$ P_t={median}(f_1,\ldots,f_B). $$

Until a all B frames are available we do not move z.

The autofocus begins with a configurable big step (s_0). If focus
decreases beyond a fraction for enough evaluations

$$ \bar{P}_t < \bar{P}_{t-1}(1-\epsilon), $$

we treat it as a overshoot, reverses direction, and
reduces the step

If focus falls below a configurable fraction (\rho) of the best observed
focus,

$$ \bar{P}_t < \rho P_{\text{best}}, $$

the autofocus treats focus as lost and restores the biggest step.

Fixed the smoothing weights originally used
min(1, n - 1) as their denominator, which I corrected to
max(1, n - 1).

I also added asyncio event loops for scan and autofocus workers.

Intensity-based scan Z

To get a correct Z position for scanning, I sample the mean image
intensity over a Z range:

$$ \mu(z_i)=\frac{1}{N}\sum_{p=1}^{N}I_p(z_i). $$

I then smooth the estimate and calculate its first derivative.

The implementation identifies:

$$ z_g = \underset{z}{{arg,max}}, \frac{d\mu}{dz}, $$

and the closest first-derivative zero crossing (z_0). When both are available,
the selected scan plane is

$$ z_{\text{scan}}=\frac{z_g+z_0}{2}. $$

Tracking Changes

Since there is no constant 16ms delay coming from USB port, I changed fixed waiting times to:

$$
t_{\text{ready}} =
t_{\text{command}}

• t_{\text{floor}}
• T_{\text{travel}}(d_{\max}),
  $$

where (d*{\max}) is the largest movement requested in the current iteration.
The tracking worker then waits until the camera reports an image retrieval
timestamp newer than (t*{\text{ready}}).

I splitted velocity and speed stage configuration into:

• fast and slow input
• precision positioning
• plate scanning
• tracking

Since I once collided the camera with the X stage, I added a rule for the unsafe combination of low Y and high Z.

Recording pipeline

I removed image encoding and I/O with disk from the camera acquisition callback
The resulting pipeline is:

camera acquisition thread
    → frame processing
    → handoff queue
    → handoff thread
    → shared-memory or standard thread queue on windows
    → image saver worker
    → TIFF file

Finally, each operation now gets each own worker abd runs independently and UI is scheduled with kivy Clock:

• camera acquisition
• image handoff and saving
• tracking
• live autofocus
• plate scanning
• multi-plate orchestration
• go-to stage movement

[iliasoroka1]

AI-Assisted Safety, Reliability, and Edge-Case Audit

Scope

This part of the pull request contains the changes produced during the
AI-assisted audit and the follow-up fixes made after testing interactive stage
control. The main audit commit was assisted by gpt-5.6-sol.

The audit did not primarily add new experimental algorithms. Its purpose was to
make the existing scanning, autofocus, tracking, recording, DAQ, and hardware
control paths fail safely under malformed input, device errors, worker failure,
shutdown, and unusual image data.

The most important distinction is:

• the direct security change is the removal of executable DAQ script parsing;
• the remaining work is mainly hardware safety, concurrency safety, recording
  integrity, deterministic cleanup, portability, and edge-case handling.

Code and Test Map

Area	Production code	Main tests
DAQ parsing and safe outputs	glowtracker/DAQ_control.py	tests/test_daq_parser.py
Stage polling, jogging, and emergency stop	glowtracker/Zaber_control.py, glowtracker/GlowTracker.py	tests/test_stage_position_poller.py
Recording integrity and failure handling	glowtracker/image_saver.py, glowtracker/GlowTracker.py	tests/test_image_saver.py
Camera frame ownership	glowtracker/Basler_control.py	tests/test_camera_control.py
Image type normalization	glowtracker/image_utils.py, glowtracker/Microscope_macros.py	tests/test_image_utils.py, tests/test_tracking.py
Worker lifecycle and cancellation	glowtracker/runtime_control.py, glowtracker/scan.py, glowtracker/MacroScript.py, glowtracker/GlowTracker.py	tests/test_runtime_control.py, tests/test_macro_stop.py, tests/test_scan_control.py
Shared-memory portability	glowtracker/SharedMemory/	tests/test_shared_memory.py
Autofocus regression coverage	glowtracker/AutoFocus.py	tests/test_autofocus.py

Secure DAQ Script Parsing

Removal of eval

The previous implementation of
DAQControl.parseTextScript() assembled the
user-entered text into a Python dictionary literal and evaluated it with
eval. Even with a restricted globals dictionary, this treated configuration
text as executable Python and made the parser's security boundary difficult to
reason about.

The audited implementation parses the text with ast.parse(..., mode="eval")
and recursively converts only explicitly allowed syntax through
DAQControl._parseScriptNode().

The accepted syntax is limited to:

• dictionaries;
• lists and tuples;
• string, integer, and floating-point constants;
• the names mode, frame, time, on, and off; and
• unary + or - applied to numeric values.

All other AST nodes are rejected. Function calls, attribute access, imports,
comprehensions, indexing, lambdas, and arbitrary expressions cannot reach
execution.

Conceptually, the boundary is:

expression = ast.parse(script, mode="eval")
parsed = DAQControl._parseScriptNode(expression.body)
validated = {
    trigger: DAQControl._validateScriptCommand(command)
    for trigger, command in parsed.items()
}

Command validation

DAQControl._validateScriptCommand()
accepts only:

[off]
[on, voltage]

It additionally enforces that:

• trigger keys are numeric but not Boolean;
• trigger keys are finite and non-negative;
• frame-mode triggers are integers;
• the mode entry contains exactly one supported mode;
• voltage is numeric and finite; and
• voltage lies within the supported 0.0 to 4.95 V range.

Duplicate dictionary keys are detected during AST traversal instead of being
silently overwritten by normal Python dictionary construction.

The active sequence is replaced only after the complete script has parsed and
validated successfully. Invalid input therefore does not partially mutate or
erase a previously valid sequence.

Regression coverage

tests/test_daq_parser.py covers:

• valid frame scripts;
• fractional triggers in time mode;
• sorting of commands;
• invalid scripts preserving the active sequence;
• executable and unsupported syntax;
• zero-valued exterior constants;
• voltage range validation; and
• DAQ safe-off behaviour.

Fail-Safe DAQ Output Handling

DAQControl.safe_off() explicitly drives both
DAC0 and DAC1 to zero. It first attempts one combined feedback operation. If
that fails, it falls back to zeroing each channel separately so that a failure
on one output does not prevent an attempt on the other.

The method also clears the running sequence state and resets the software's
current-voltage state.

DAQControl.close() performs safe-off before
closing the device connection. Connection cleanup is in finally blocks so
that an output or close exception cannot leave the application believing the
device is still connected.

DAQControl.reset() similarly wraps the device reset so that safe_off() is
still called if restoring factory defaults fails.

The application also calls safe_off() when:

• recording cleanup cannot reset the DAQ normally;
• application shutdown finds workers that cannot be joined safely; or
• another hardware failure requires a conservative fallback.

This changes the DAQ policy from "close the handle and assume the output is
safe" to "attempt to establish a known zero-output state, report failure, and
then release the handle."

Stage I/O Serialization and Position Polling

Problem addressed

Before the audit, UI updates, key handlers, tracking, safety checks, and other
workers could all request stage coordinates directly. Hardware reads can block,
and overlapping asynchronous or synchronous reads can conflict with each
other. This was particularly visible when pressing or releasing movement keys
while a position read was in progress.

Dedicated position poller

Stage.start_position_poller() starts a
named StagePositionPoller worker. The worker is implemented by
Stage._position_poll_loop().

The poller:

• serializes hardware coordinate reads through _position_read_lock;
• stores the most recent position and update time under a condition lock;
• changes its polling interval when Y/Z collision monitoring is active;
• wakes immediately when a jog command requires attention; and
• stops when teardown or disconnect begins.

Stage.get_cached_position() exposes a
copy of the last position in the requested units and can reject stale cache
entries using max_age.

The UI coordinate paths in GlowTrackerApp.stage_stop(),
GlowTrackerApp._keyup(), and GlowTrackerApp.update_coordinates() now read
this cache instead of performing hardware I/O on the Kivy thread.

Queued jog commands

Interactive commands enter the stage through:

• Stage.request_start_move(); and
• Stage.request_stop().

These functions enqueue commands and wake the poller rather than blocking the
keyboard, controller, or button callback.

Stage._process_jog_commands() executes
the queued commands in order on the poller worker. Per-axis generation counters
prevent duplicate stops and preserve the ordering of a release followed quickly
by another press.

Interactive stop uses Zaber's no-response stop command in
Stage._stop_jog_no_response(). The input callback therefore does not wait for
a normal request/response round trip before it can return.

The follow-up keystroke fixes in
glowtracker/GlowTracker.py:

• route key-down through GlowTrackerApp.request_jog();
• route key-up through GlowTrackerApp.request_stage_stop();
• ignore key-up events unrelated to stage movement;
• remove direct get_position() calls from input handlers;
• remove direct acceleration changes from the UI thread; and
• remove speculative coordinate extrapolation based on key velocity.

Displayed coordinates now come from actual polled stage data rather than an
estimate calculated from a presumed frame duration.

Position-poller regression tests

tests/test_stage_position_poller.py
verifies that:

• position reads occur on StagePositionPoller;
• two-axis stages expose a usable zero-Z coordinate;
• a failed safety read while Y is jogging causes a stop;
• Y collision safety uses the polled coordinates;
• X-only jogging does not unnecessarily activate Y/Z collision polling;
• stop and subsequent start requests do not block their caller;
• queued stops execute before later coordinate reads;
• key-down does not call blocking movement APIs directly;
• key-up uses the queued stop path; and
• UI coordinate methods only use the stage cache.

Emergency Stop and Hardware Teardown

Stage emergency stop

Stage.emergency_stop():

1. clears queued jog commands;
2. attempts connection.stop_all(wait_until_idle=False);
3. falls back to stopping the individual axes if stop-all fails; and
4. resets the software movement state and recorded velocities in finally.

Jogging also fails closed. If the position required for Y/Z collision safety
cannot be read, the poller calls emergency_stop() instead of continuing
movement without a verified position.

Cancellable stage setup

Stage.home_stage() and Stage.on_connect() accept a cancellation event.
Cancellation is checked between homing, range setup, initial movement, and
device-idle waits.

Connections.connectStage() in
glowtracker/GlowTracker.py performs setup in a
background worker, keeps the movement controls disabled until setup completes,
and starts the position poller only after a valid initial position has been
obtained.

Connections.disconnectStage():

• signals setup cancellation;
• disables stage controls;
• cancels and waits for active go-to movement;
• cancels periodic coordinate display updates;
• triggers an emergency stop;
• waits for the setup thread; and
• disconnects only after those operations are inactive.

If the go-to or setup worker does not stop within the deadline, disconnection is
deferred and the UI connection state is restored.

Stage.disconnect() itself stops the position poller before closing the
connection and clears all axis and device references afterward.

Coordinated application shutdown

GlowTrackerApp.stop_active_workers()
provides a central shutdown sequence for:

• go-to movement;
• scan and multi-plate workers;
• macro execution;
• tracking;
• live focus;
• recording acquisition;
• live-view acquisition; and
• image-saving workers owned by recording.

The method requests cancellation first, issues an emergency stage stop when
motion workers remain active, and then joins workers against one shared
deadline. It returns the names of workers that are still alive.

GlowTrackerApp.graceful_exit() sets the
global _hardware_teardown guard before stopping workers. If any worker remains
active, it does not proceed with normal hardware disconnection; it instead
emergency-stops the stage and zeros the DAQ outputs.

This prevents the application from closing a hardware connection while another
thread may still be using it.

Connections.disconnectCamera() follows the same policy: camera disconnect is
cancelled and the UI is restored if acquisition, scanning, focus, tracking, or
recording workers cannot be stopped.

Managed Background Movement

ManagedStageMove replaces an unmanaged
go-to thread with a single-flight movement object.

It:

• refuses a new move while another move is alive;
• refuses movement when hardware teardown has started;
• stores a cancellation event;
• calls stage.emergency_stop() when cancellation is requested;
• suppresses success callbacks after cancellation or teardown; and
• provides bounded wait() and is_active() operations.

The GoToControls widget in
glowtracker/GlowTracker.py delegates its
movement, stop, wait, and active-state operations to this class.

tests/test_runtime_control.py exercises
single-flight behaviour, cancellation, teardown rejection, success callbacks,
controller deadband, and synchronized live-focus graph updates.

Scan and Macro Cancellation

Scan lifecycle

CenterRadiusFromThreePoints stores explicit references
to its scan and multi-plate threads. New scans are refused while either worker
is already running.

Each workflow receives a generation number. Delayed Kivy callbacks check that
generation before modifying camera, plate, tracking, or recording state. A
callback left over from an older run therefore cannot restart work after a
stop or new run.

CenterRadiusFromThreePoints.request_shutdown():

• invalidates the active generation;
• marks teardown requested;
• sets both scan stop flags;
• releases the track wait event; and
• emergency-stops the stage if a scan worker is active.

CenterRadiusFromThreePoints.wait() joins both scan
workers against an optional shared timeout.

Fixed sleeps in the scan loop were replaced by _wait_or_stop(), which checks
stop state in short intervals. The scan can therefore respond during settling
instead of waiting for the full delay.

The audited scan additionally:

• verifies every tile movement, not only the first tile;
• stops if a recentering movement is refused;
• does not begin tile scanning when Z estimation fails;
• supports cancellation during the Z sweep;
• aborts if live acquisition does not stop before scan camera configuration;
• avoids restoring camera settings during global hardware teardown; and
• restores the precise motion profile in finally.

tests/test_scan_control.py specifically verifies
that a failed Z sweep cannot start XY tile scanning.

Macro lifecycle

MacroScriptExecutor.stop() now sets the
termination flag and can optionally wait for its execution thread.

MacroScriptExecutor.wait() safely handles
missing, completed, and current-thread cases before joining.

Macro wait commands no longer use one long time.sleep(). They sleep in
short intervals and recheck the termination flag, allowing a stopped macro to
exit promptly instead of blocking hardware teardown for the entire requested
wait duration.

tests/test_macro_stop.py covers interruption of a
running macro wait.

Camera Frame Ownership

The Basler API exposes image data through a grab-result object whose storage may
become invalid or reused after Release().

Camera.retrieveGrabbingResult() now
creates an owning NumPy copy before releasing the result:

img = np.array(grabResult.Array, copy=True)

The grab result is released from a finally block whether the grab succeeds or
fails. Boolean timeout results are not treated as grab-result objects.

This guarantees that the image passed to tracking, display, recording, or
another thread no longer depends on the lifetime of the camera SDK object.

tests/test_camera_control.py verifies:

• use of the real pypylon base class;
• ownership of returned image memory after release;
• release of unsuccessful grab results;
• idle behaviour when the camera is not grabbing; and
• parsing of camera feature files.

Recording Integrity and Saver Failure Handling

Frame acknowledgement model

SaveAcknowledgements associates each
captured frame with:

• its coordinate row;
• the image channels that must be written;
• completion/failure state; and
• a monotonically increasing frame index.

For a single-colour frame, one saver acknowledgement completes the frame. For a
split dual-colour frame, both the main and minor channels must be acknowledged.

Coordinate rows are flushed strictly in frame order and only after all expected
channels have been saved. A failed frame has no coordinate row, but later
successful frames can still be flushed once the failed index has been resolved.

This prevents the coordinate log from claiming that an image exists when the
corresponding TIFF write failed.

Atomic file publication

save_worker() writes each image to a
temporary path such as:

frame.part.tiff

After tifffile.imwrite() completes, os.replace() atomically publishes the
final filename. If writing fails, the partial file is removed.

This avoids leaving a normal-looking final TIFF containing incomplete data.

Split dual-colour frames receive deterministic suffixes:

frame-main.tiff
frame-minor.tiff

Explicit saver status channel

Savers report one of:

("saved", frame_index, channel, "")
("failed", frame_index, channel, error)

RecordButton._saveStatusLoop() consumes
these statuses, updates acknowledgements, and detects a worker that exits
without reporting completion.

RecordButton._saveHandoffLoop() moves
frames from the small acquisition-side queue into the process or thread saver
queue. Queue overflow, transport exceptions, and an already failed saver all
resolve the affected acknowledgement as failed.

The recording acquisition condition includes the saver failure event. A disk or
worker failure therefore stops further capture instead of continuing to collect
frames that cannot be written.

Bounded cleanup

RecordButton._cleanupImageSaver()
coordinates shutdown of:

• the handoff thread;
• the saver process or saver threads;
• the saver-status monitor;
• the shared-memory manager;
• pending acknowledgements; and
• inter-process queues.

Each stage has a timeout. A stuck saver process is first terminated and then
killed if the platform supports it. Workers that still cannot be stopped are
retained in _abandonedSavers so their resources are not destroyed while they
may still be in use.

close_file_with_timeout() prevents an
indefinitely blocked coordinate-file close from freezing recording cleanup.

Cleanup is protected by _recordingCleanupLock, making repeated stop callbacks
idempotent. This matters because stop can be reached from the button, the
acquisition thread, automatic frame completion, a saver error, or application
shutdown.

Recording regression tests

tests/test_image_saver.py covers:

• bounded coordinate-file close;
• close exceptions;
• split-channel suffixes;
• TIFF write failure;
• atomic-publication failure and partial-file removal;
• all-channel acknowledgement before coordinates are written;
• preservation of frame order;
• omission of coordinates for failed frames;
• continued coordinate output after a resolved failure; and
• inclusion of DAQ voltage only through the acknowledgement path.

Image Type and Tracking Edge Cases

Image normalization

normalize_image() establishes a common
floating-point range for processing:

• Boolean images become 0.0/1.0;
• integer images are scaled using the full range of their dtype;
• floating-point images already in [0, 1] are preserved;
• other finite floating-point images are min/max normalized;
• NaN and infinity are replaced after checking that finite data exists; and
• empty or entirely non-finite images are rejected.

For unsigned integer type (T), the normalization is

$$
I_{\text{norm}}

\frac{I-\min(T)}{\max(T)-\min(T)}.
$$

This fixes the previous assumption that any value above 1 should simply be
divided by 255, which produced incorrect masks for 12-bit or 16-bit cameras.

effective_max_brightness() preserves the legacy
configured maximum of 255 for 8-bit images but expands it to the dtype maximum
for wider unsigned camera images.

prepare_texture_data() preserves native
uint8, uint16, and float32 texture formats and safely converts unsupported
types.

Tracking masks and centroids

create_mask() uses the shared
normalization helper and guards gamma correction against means of exactly 0 or
1.

find_CMS() rejects:

• empty masks;
• completely black masks; and
• completely non-zero masks.

These cases contain no meaningful foreground centroid and previously could
produce invalid region selection.

Tracking brightness ranges now validate that the minimum does not exceed the
effective maximum. Diagnostic overlay coordinates are also calculated from the
actual cropped image bounds.

tests/test_tracking.py covers crop boundaries,
coordinate transforms, dark- and bright-background masks, 8/16-bit
equivalence, centroid selection, uniform masks, movement-sign conventions, and
diagnostic output.

Reversal-Detection Edge Cases

ReversalDetector.detectReversal() now
normalizes its input and returns False for malformed, empty, or one-point
trails.

The number of history vertices used for velocity estimation is clamped to a
valid range:

$$ 2 \leq N_{\text{velocity}} \leq N_{\text{body}}. $$

Stationary trails are handled before angle calculation. Both the tail-to-head
vector and estimated velocity must have non-zero magnitude.

For valid vectors, the signed angle is calculated from scalar 2D cross and dot
products:

$$
\theta

\operatorname{atan2}
\left(
x_1y_2-y_1x_2,,
x_1x_2+y_1y_2
\right).
$$

Reversal is detected symmetrically using

$$ |\theta| > |\theta_{\text{threshold}}|. $$

DAQControl.updateReversalDetection() also avoids sending a duplicate voltage
command when the desired output has not changed.

The reversal tests in
tests/test_daq_parser.py cover short trails,
stationary trails, forward motion, and reverse motion.

Focus-Sweep and Scan Edge Cases

DepthOfFieldEstimator.takeCalibrationImages() and
IntensitySweeper.sweep() now reject
sample counts below one.

For exactly one sample:

• the Z step is zero;
• the sample is taken at the current or midpoint position; and
• no division by numImages - 1 occurs.

The sweep no longer issues an extra movement after capturing the final sample.
Movement return values are checked, and the starting position is restored from
finally when the operation has not been explicitly cancelled.

IntensitySweeper.sweep() accepts a stopRequested callback so scan shutdown
can interrupt a long Z sweep.

tests/test_autofocus.py locks in the configured
smoothing-weight range and verifies that focus history updates only after the
frame buffer is complete.

Shared-Memory Portability and Validation

The original shared-memory counters used the external atomics package and
were described as lock-free.

SharedAtomicCounter now uses
multiprocessing.Value("Q", ..., lock=True) from the selected multiprocessing
context. load, store, and add explicitly acquire the value lock.

This:

• removes a platform-sensitive native dependency;
• works with the same spawn/forkserver context as the saver process;
• makes the synchronization guarantee explicit; and
• avoids incorrectly describing the queue as lock-free.

SharedMemoryQueue.put()
validates the complete schema before touching shared memory:

• keys must match the declared fields exactly;
• NumPy shapes must match;
• NumPy dtypes must match; and
• scalar fields must be numeric.

Schema mismatches therefore fail before partially updating a queue slot.

tests/test_shared_memory.py starts a spawned
child process and verifies that a frame can be transferred through the audited
queue implementation.

Packaging, CI, and Documentation

The audit aligned the project metadata and automated tests:

• .python-version recommends Python 3.12;
• pyproject.toml declares Python 3.11 through 3.13;
• runtime dependencies use bounded version ranges;
• opencv-python-headless is used for non-GUI OpenCV functionality;
• tifffile is declared directly;
• the external atomics dependency is removed;
• a test optional dependency installs pytest;
• uv.lock records the resolved environment;
• .github/workflows/install_package.yml
  installs the package with test dependencies and runs pytest on Linux,
  macOS, and Windows for Python 3.11, 3.12, and 3.13; and
• README.md and BUILD.md document the supported
  Python versions and the uv-based setup.

Validation

The final integrated branch was validated locally with:

uv run pytest -q

Result:

79 passed

The suite covers the security boundary, hardware-control helpers, worker
lifecycle, recording failure paths, image ownership, image dtype handling,
tracking calculations, scan cancellation, shared-memory transport, and the
autofocus regressions identified during the audit.

Resulting Safety Model

The audited application now follows these general rules:

1. Configuration text is parsed as data, not executed as code.
2. Hardware outputs are explicitly moved toward a known safe state before
   connections are released.
3. UI callbacks do not perform blocking stage reads or movement stops.
4. Stage position reads and interactive jog commands are serialized.
5. Failure to verify a safety-critical stage position causes movement to stop.
6. Hardware is not disconnected while known workers are still active.
7. A recorded coordinate row is committed only after its image data is
   confirmed on disk.
8. Partial image files are not published under final filenames.
9. Image processing accounts for actual dtype and invalid/uniform data.
10. Long-running scans, macros, focus sweeps, and acquisition workers have
    explicit cancellation and bounded shutdown paths.