I was running some experiments in which the evaluator file did not directly implement a hard timeout for a given program, thus a program could run for an arbitrary amount of time. The code in process_parallel.py (L532-778) first finds a given future that is done, and then proceeds to a try-except block to recover the result. We only enforce the timeout with a future.result(timeout=timeout) call, but this is only done to futures that were completed, i.e., the ones with future.done() == True. I think this only handles futures that are done but are stuck in returning their results, but not futures that haven't been completed and are stuck or running for more than the timeout budget. In my experiments, the timeout was 60s, but some futures took over 1500s to complete, which leads to all sorts of weird situations. I know this could be solved by implementing a hard timeout in the evaluator file, but I think it would be nice if this was also enforced by openevolve itself. If this makes sense, I can make a PR later.
I was running some experiments in which the evaluator file did not directly implement a hard timeout for a given program, thus a program could run for an arbitrary amount of time. The code in
process_parallel.py(L532-778) first finds a given future that is done, and then proceeds to a try-except block to recover the result. We only enforce the timeout with afuture.result(timeout=timeout)call, but this is only done to futures that were completed, i.e., the ones withfuture.done() == True. I think this only handles futures that are done but are stuck in returning their results, but not futures that haven't been completed and are stuck or running for more than the timeout budget. In my experiments, the timeout was 60s, but some futures took over 1500s to complete, which leads to all sorts of weird situations. I know this could be solved by implementing a hard timeout in the evaluator file, but I think it would be nice if this was also enforced by openevolve itself. If this makes sense, I can make a PR later.