Ensure workable application health probes

[jcpunk]

With ERS having hung last week, I've updated the microservices to have trivial naive health probes. They aren't fancy, but should improve recovery under stuck conditions.

[MRiganSUSX]

Thanks @jcpunk.
Here are some comments.

[MRiganSUSX]

For inspiration, the fixes are applied in https://github.com/DUNE-DAQ/microservices/tree/mrigan/fix_health_probes.

[jcpunk]

Changes merged in

[MRiganSUSX]

Thanks @jcpunk for the quick fixes.

I tried test-deploying these in their own namespaces (ers-test and opmon-test).
It did not work out of the box (ofc modding the yamls to target the test env correctly).

After several iterations with AI I ended up with https://github.com/DUNE-DAQ/microservices/tree/fix/health-probes-from-patt, and especially the commit 93aeecd which is fixing 3 things that were problematic. The description has more details:

Fix health server: raw socket accept loop, remove premature shutdown
Three issues fixed in both dbwriters:

1. Replace HTTPServer/serve_forever() with a raw socket accept loop.
   serve_forever() uses EpollSelector which cannot acquire the GIL while
   the Kafka subscriber thread holds it, so accept() is never called and
   probes time out.

2. Remove stop_health_server() from the finally block after sub.start().
   sub.start() is non-blocking — it returns immediately after starting
   Kafka's internal threads, so the finally fired instantly and closed
   the listening socket before any probe arrived.

3. opmon: use influx.ping() instead of influx.get_list_database() in the
   readiness check — lighter weight and purpose-built for health checks.

I'm not an expert, the only thing I can confirm is that it did not work before, and it does work with these changes.

Thoughts on these? Should we merge some/all into your branch?

[MRiganSUSX]

(the test deployments are still there if you want to play)

[jcpunk]

I think I merged your branch into mine, did I miss something?

[MRiganSUSX]

This is yet another branch. The first one was a code review, this one is a result of actually testing deploying it.

[jcpunk]

Got it, branch merged in.

[MRiganSUSX]

Run a test deployment and worked as expected with probes returning status.