We currently cap the maximum duration between recovery probing for a block at 16 rounds (160 minutes). This seems to result in massive synchronization of recovery probing every 160 minutes. (Perhaps due to blocks that are uncertain/down right from the first round?) This massive spike in probes sent results in a corresponding reduction in the responsive rate of periodic probes, which is probably not a big deal since the adaptive probing should fix things up, but it does mean we're wasting probes, and annoying people more than we need to.
See https://charthouse.caida.org/@Jt for an example of these spikes and corresponding drop in responsive periodic probes.
https://github.com/CAIDA/trinarkular/blob/master/lib/trinarkular_prober.c#L124 and https://github.com/CAIDA/trinarkular/blob/master/lib/trinarkular_prober.c#L126 are where this problem stems from.
My suspicion is that if we simply introduce some randomness in this number we'll probably solve this problem, but there may be a more thoughtful solution, like reconsidering how recovery probing works in general.
We currently cap the maximum duration between recovery probing for a block at 16 rounds (160 minutes). This seems to result in massive synchronization of recovery probing every 160 minutes. (Perhaps due to blocks that are uncertain/down right from the first round?) This massive spike in probes sent results in a corresponding reduction in the responsive rate of periodic probes, which is probably not a big deal since the adaptive probing should fix things up, but it does mean we're wasting probes, and annoying people more than we need to.
See https://charthouse.caida.org/@Jt for an example of these spikes and corresponding drop in responsive periodic probes.
https://github.com/CAIDA/trinarkular/blob/master/lib/trinarkular_prober.c#L124 and https://github.com/CAIDA/trinarkular/blob/master/lib/trinarkular_prober.c#L126 are where this problem stems from.
My suspicion is that if we simply introduce some randomness in this number we'll probably solve this problem, but there may be a more thoughtful solution, like reconsidering how recovery probing works in general.