[Tests] Exception: Couchbase did not become ready after 30 attempts -- Bind for 0.0.0.0:8091 failed: port is already allocated

[amotl]

Problem

While refactoring the test suite with GH-79, we are observing this error on our workstation.

FAILED tests/warehouse/db/test_couchbase.py::test_couchbase_source_local[duckdb] - Exception: Couchbase did not become ready after 30 attempts
FAILED tests/warehouse/db/test_couchbase.py::test_couchbase_source_local[clickhouse+native] - docker.errors.APIError: 500 Server Error for http+docker://localhost/v1.54/containers/a4277ee2ccab0b4f5d90717c6f4085d1a089bff8a3ee57f7ecb56e91b1771e43/start: Internal Server Error ("failed to set up container networking: driver failed programming external connectivity on endpoint dazzling_faraday (eeb2236b1c1c50462589131a9b09e247a0126e73dd3879193c60dc80d0588932): Bind for 0.0.0.0:8091 failed: port is already allocated")
ERROR tests/warehouse/db/test_kafka.py::test_kafka_to_db_incremental[clickhouse+native] - RuntimeError: Failed to start container: kafka

Details

tests/warehouse/db/test_couchbase.py:27:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
tests/util/container/impl/couchbase.py:31: in start
    self._wait_for_couchbase()
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <tests.util.container.impl.couchbase.CouchbaseContainer object at 0x162adaad0>

    def _wait_for_couchbase(self):
        """Wait for Couchbase to be ready."""
        import requests

        port = self.get_exposed_port(8091)
        base_url = f"http://{self.get_container_host_ip()}:{port}"

        max_attempts = 30
        for i in range(max_attempts):
            try:
                response = requests.get(f"{base_url}/pools", timeout=2)
                if response.status_code == 200:
                    return
            except Exception:
                pass
            time.sleep(2)

>       raise Exception(f"Couchbase did not become ready after {max_attempts} attempts")
E       Exception: Couchbase did not become ready after 30 attempts

tests/util/container/impl/couchbase.py:64: Exception

[amotl]

There is a comment in test_couchbase_source_local, which educates why this container driver is special, because it doesn't use ephemeral port assignments, but fixed ones.

# NOTE: This test requires local Couchbase Server to be stopped first,
# as it uses 1:1 port mapping (8091, 11210, etc.) to avoid SDK connection issues.

[amotl]

I think the solution here is to use a standard couchbase testcontainer implementation instead of the custom one.

• feat(modules): add Couchbase module testcontainers/testcontainers-python#790

Until it is included into upstream testcontainers, it should go into tests/util/container/impl.

[hampsterx]

i can take this one if you havent started already

[amotl]

Hi again.

My original post was just a note and doesn't hurt me too much. CI/GHA currently works well and is stable, this is mostly what I am caring about. However, maybe others who want to run the test suite locally are also annoyed about any flaws here. The flaws are especially more annoying if you run the test suite more than once, because of stale Couchbase containers not always being torn down.

If you want to tackle this, go ahead, thanks! However, the task could be daunting, as it will likely include running the test suite over and over again in different slices, mixed with funny experiences about testcontainer behaviours.

In general, about coordinating work sharing, we could use GitHub issue assignments to signal if someone starts working on something?

With kind regards,
Andreas.