Modify Ad hoc task queue check (tool_task_adhocqueue) to be a bit more intelligent

[dmitriim]

Currently tool_task_adhocqueue check simply gets the oldest unprocessed task and alarms if it over the threshold value.

CRITICAL: Ad hoc task queue: Oldest unprocessed task is 15 hours 50 mins, which is more than 4 hours

In real life this may not actually mean an issue. On big sites where long running autobackup tasks can be stacked in a queue, but eventually get processed which takes time this generates a false alarm.

Should we instead look at the total number of tasks in the queue?
Or maybe we should check if the specific old task has a queue and the queue is getting processed?

[dmitriim]

tool_task_adhocqueue is a core check if we would like to improve it we would either create a tracker or replace it with the one form tool_heartbeat and add the core one to list of suppressed checks https://github.com/catalyst/moodle-tool_heartbeat/blob/MOODLE_39_STABLE/classes/checker.php#L333

[brendanheywood]

As mostly discussed elsewhere:

1. generally we want to use time / latency metrics because they are a better indicator of health than count. eg a large site may process 10k tasks an hour, so alerting if the task count > 1k or any other default number is not going to work, and we want to avoid needing to tune things on each site.

2. so things like a latency measure for backups is actually still the best, eg if backups are so far behind that they are a week late thats actually pretty bad and cron needs more scale to keep that within a reasonable timeframe.

3. We do however want to improve that check so that it is per-task type. Both exposing what is the slowest task in the summary and also showing all the slow task types in the details.

4. on top of that we'd want the ability to have per task type thresholds, so for example we might never want a forum email task to be processed later than 4 hours after it should have been, and backups can be up to 1 day old for example.

I think this one is still best done in core and we can back port, the lest replacing of tasks we do in heartbeat the better.

I've made: https://moodle.atlassian.net/browse/MDL-88496