Unify Provisioner implementation synchronization semantics

[comps]

Currently, each Provisioner uses very different synchronization approaches:

• TestingFarmProvisioner has daemon=True threads that silently die in the background even after .stop()
• SharedVirtProvisioner has one non-daemon thread with proper threading.Event waiting, but has very over-complicated creation/joining of the one thread (via a threading.Semaphore) depending on when it is / is not needed, to prevent (well, limit) race conditions with its use
• PodmanProvisioner just uses a threading.Condition and creates containers inside .get_remote(), not in any background thread
• TempVirtProvisioner combines threading.Event with threading.Condition and a collections.deque for the thread returns

It would be great to at least converge on some common fool-proof solution for all/most of these.

For example:

1. Standard daemon=False threads with threading.Event propagation to lower-level functions, to be done ie. in testingfarm/api.py
2. threading.Condition to be always used for synchronization (as a generic lock and also .wait_for()), to allow .stop() waking up a blocking .get_remote() across all implementations
   • (also .stop() on .get_remote() should always be an exception, not a random None from a blocking call)
3. A simple FIFO collections.deque for passing returns/exceptions from threads, in combination with the threading.Condition

Not all provisioners need to use all these, but it also shouldn't be some random threading.Semaphore coupled to the logic.

[comps]

(random notes from a TODO list)

- redo the complex thread spawning/exiting in SharedVirtProvisioner
  - just have one long-running thread, started in .start(), stopped in .stop()
  - send work to it via a Condition:
        cv = threading.Condition()
        pending = 0  # or whatever integer state
        done = False
        # Worker
        with cv:
            while True:
          cv.wait_for(lambda: pending > 0 or done)
          if done:
              break
          # use pending
          pending = 0
        # Add work
        with cv:
            pending += 1000
            cv.notify()
        # Signal exit
        with cv:
            done = True
            cv.notify()

- abort=None for TF api.Reserve and PipelineStreamer
  - if specified as a threading.Event, abort the reservation ASAP when it triggers,
    cancelling any in-progress TF Requests
    - works only while preparing the reservation, NOT after it is handed
      to the user
  --> allows TestingFarmProvisioner to use non-daemon threads

- add shutting_down=True/False or running=True/False logic to more places
  - ie. if a new remote is to be added to self.remotes (or so), check that,
    inside self.lock, the class instance isn't being shut down
  - theoretically:
    - remote-creating thread reserves a new system and it tries to self.lock
      before appending it to the list of remotes
      - EVEN THOUGH append is atomic
    - stop() gets self.lock first, it releases all remotes
    - then the thread runs, appends on remote, never to be released
  - the fix
    - remote-creating thread first checks, AFTER getting self.lock,
      if we're still running=True and if not, self-releases the remote
    - if we *are* running=True, it means stop() wasn't called yet, and
      so releasing is possible in the future
  - alternatively, instead of running=True/False, set self.remotes=None
    if not running, and [] / set() if running