Change relock_tracking to setup_tracking in uxm_relock

[tpsatt]

Lately, we've been made aware of the fact that tracking can degrade over time. Right now, we have an alert which ROCs flag to SMuRF Tsars, who then run setup_tracking to fix the issue. Best case scenario, it can be a bit difficult for SMuRF Tsars to take time to do this, and to find time in the platform's schedule. Worst case scenario, we miss this and have degraded performance without realizing it.

Since we run uxm_relock each day, running setup_tracking each time should fix the issue. We lose a few minutes from added execution time, but we save the overhead of identifying and fixing the degraded tracking.

It's not entirely clear why this is happening in the first place; we don't expect drifts to the FR put out by the RTM. But if this PR fixes the issue, it may not warrant further investigation given the bandwidth constraints of the relevant parties.

[dpdutcher]

I'm not sure that we want to do this. For example, this issue raises the exact opposite problem of wanting frac_pp to be even more persistent: #484

At a bare minimum, this needs to be an option, and one that is probably set to default False. The number of systems that have required running setup_tracking_params to fix is a definite minority, and we risk mucking the majority up by fiddling with their frac_pp settings every day.

If we want to implement this, we need to make the frac_pp estimation more robust than it currently is, possibly by making the underlying pysmurf algorithm more robust, or by tweaking the sodetlib implementation of it.

[msilvafe]

I agree we don't want this. If the tracking parameters are changing regularly this is an indication of some system failure which shouldn't go unnoticed.

[tpsatt]

I see your points about not ignoring drifts to the tracking parameters. But over the past few weeks, I've had to rerun setup_tracking_params on many LAT systems to recover their noise performance. Is there someone who has the bandwidth to look into this? I assume not, which is why I had suggested this crude fix.

[msilvafe]

I see your points about not ignoring drifts to the tracking parameters. But over the past few weeks, I've had to rerun setup_tracking_params on many LAT systems to recover their noise performance. Is there someone who has the bandwidth to look into this? I assume not, which is why I had suggested this crude fix.

Let's discuss on updated R+D Ops call. I'll try to make it.

[dpdutcher]

My suggestion: Make this an option set to False by default and also include an option to re-setup the phase delay, default False.

[msilvafe]

If we do this (auto retry phase delay and relock tracking) I would like us to figure out how to publish that this retry is triggered and have it publish to grafana so we can track it and alarm on it.