[Bug] follow is affected by large number of jobs

[alippai]

Describe the bug

When there is a large number of jobs (thousands), then the pueue follow TASK_ID periodically prints text regarding large pueue network messages.

I'm not sure it's about the follow or if it's an internal healthcheck or actually written to the logfile as well.

This might be obvious from the code, but also I'll be able to create a small example later.

Steps to reproduce

NA yet

Debug logs (if relevant)

No response

Operating system

Linux

Pueue version

v4.0.1

Additional context

No response

[Nukesor]

Could you provide some log output? From the description alone, I have no idea where to begin debugging :D

[alippai]

@Nukesor: :D
Sorry, I know it was brief, I just cleaned the jobs and didn't saw the message again.

Now I have it again:

pueue follow JOB_ID
WARN pueue_lib::network::protocol: Client is sending a large payload: 23183073 bytes.
WARN pueue_lib::network::protocol: Client is sending a large payload: 23183073 bytes.
WARN pueue_lib::network::protocol: Client is sending a large payload: 23183073 bytes.
WARN pueue_lib::network::protocol: Client is sending a large payload: 23183073 bytes.

The message is printed every few seconds. I have a feeling because of the large number of jobs not cleared in pueue. It's not part of the log file, likely it's printed by the pueue follow command

[Nukesor]

Could you give some more information, like number of jobs, amount of log output?
Do you use read_local_logs: false?
Does the issue also appear when calling pueue status?
Is that a full follow or just a partial follow?

Is the log output you sent exactly what you see on the terminal output? I.e. you call follow and the warnings pop up before the first bit of log output can be seen?

Also, what's your usecase :D How do you end with that many tasks in your queue?

[alippai]

I believe it's coming from the get_task(), it fetches the full state and filters after.
https://github.com/Nukesor/pueue/blob/main/pueue/src/client/commands/mod.rs#L114-L128

I'm submitting a medium size DAGs locally (about 400 jobs) every day. While pueue is not a real scheduler this way I can ensure the logs are available, tasks are retriable / cancellable by teammates. It's local to the machine, simple. It's a pragmatic compromise between setting up a full airflow/slurm/jenkins with DB or running just random jobs via cron directly in the background.

[Nukesor]

I just tested a bit. Pueue indeed becomes pretty slow when confronted with thousands of tasks. I didn't encounter this specific issue, but I can imagine that we'll run into large payload issues at some point.

Given that this is not in the current scope of Pueue, I'll put this on hold until #669 is figured out :)