OpenAI timeouts are broken, and Runner doesn't enforce them consistently

[athewsey]

Per openai/openai-python#2599, timeout settings are not actually being respected in upstream OpenAI Python SDK - and therefore also not in our OpenAI Endpoints.

This seems mostly like something that should be fixed upstream rather than worked around in LLMeter - but then I noticed the Runner does make some attempt to enforce that _invoke_n_no_wait is abandoned after N requests * Runner.timeout (for count-bound runs only)... And the Runner docstring does claim that the timeout setting is enforced (without disclaiming different run types). I'd note that even for count-bound runs, enforcing total timeout is not the same as enforcing request-level timeout.

We should try to clarify the behaviour here and improve its consistency if practical. If Runner-level enforcement is possible, it could be a nice guard against inconsistencies or errors in Endpoint implementations... But _invoke_n_no_wait is currently just a synchronous while loop of direct self._endpoint.invoke(p) so i don't think it'll be easy to add without another layer of asyncio/thread delegation.

[athewsey]

From a quick talk with Claude about potential implementations for Runner-level per-request timeout enforcement, one thing that concerns me is that apparently there's no way to safely cancel a Python thread blocked in a C-level syscall (like socket recv()) - which as I understand means:

1. Using a similar asyncio.wait_for(asyncio.to_thread(...), timeout=self.timeout) pattern to dispatch individual invocations means that any timed-out request would be correctly flagged as an error but would leave a zombie thread in the pool until the underlying connection returned something & entered back into Python bytecode.
2. If this happened a lot (which could be the case since at the moment we're trying to run tests with hundreds of clients against endpoints that are sometimes taking many minutes to respond vs the 60s we'd like) - then our thread pool would become exhausted leading the rate of requests generated by LLMeter to slow down
3. This equally apply equally to threading approaches beyond asyncio. Multiprocessing might work because processes can be killed, but the OS overhead would be really substantial and probably make LLMeter the bottleneck anyway for hundreds of clients.

The options Claude suggested were:

1. End-to-end AsyncIO where possible: If Endpoint.invoke() were an async method using an async HTTP client (httpx async, aiohttp), then asyncio.wait_for cancellation actually works because the cancelled coroutine closes the underlying socket which aborts the IO. However, it'd be a big API change and still wouldn't help much if the underlying Endpoint implementation uses synchronous IO.
2. Keep threads, size the pool large: Accepts zombies as the cost of maintaining request rate and relies on OS socket timeouts to eventually reclaim them - but for the kinds of tests I'm trying to run, tens of thousands of zombies could be on the horizon which is going to be a problem for typical OSs.
3. Keep threads, close the socket from outside: Only works for HTTP clients that expose their underlying transport, which seems like a mess to bring in to our API contract.

None of which seem particularly appealing...