Add comprehensive end-to-end integration tests with real Docker#65
Merged
Conversation
Add end-to-end integration tests for phases 0-3 (real Docker, live CoreDNS SOA query, step-ca ACME directory) and cross-world federation (PEERED mode DNS stub written to CoreDNS Corefile, NONE mode no-op). Fix two handler bugs uncovered during e2e work: - dns.py: CoreDNS started with network_mode=none then connected to 'core' network with static IP; volume mount changed ro→rw so GatewayPortalHandler can append stub zones at runtime - docker_handler.py: exec_command used demux=True but decoded .output as bytes; changed to demux=False for correct bytes output Add --run-e2e pytest flag, e2e marker, and CI e2e job (non-slow tests only, runs on ubuntu-latest with Docker pre-installed). Mark both roadmap items complete in README. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NdYtbgMhKYfk5kkw3jERWh
- Run black on test_e2e_fullstack.py and test_e2e_federation.py to fix linting failures - Add daemon.json step in CI e2e job to set default-address-pools to 192.168.128.0/18 so the runner's pre-existing networks don't overlap with the world spec's core subnet (10.0.0.0/24) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NdYtbgMhKYfk5kkw3jERWh
aidankhogg
added
documentation
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NdYtbgMhKYfk5kkw3jERWh
GitHub Actions ubuntu-latest runners have 10.0.0.0/24 reserved in Docker's IPAM (the runner host sits in the 10.x.x.x range), so any attempt to create a Docker network with subnet 10.0.0.0/24 fails with "Pool overlaps with other one on this address space". Add tests/fixtures/e2e-spec.yaml — identical to minimal.yaml except the core network uses 172.20.0.0/24 and all core-IP services use 172.20.0.x addresses. Both e2e test files now load this fixture instead of examples/minimal.yaml. The production example is unchanged. Also removes the non-functional daemon.json address-pool workaround from the CI e2e job. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NdYtbgMhKYfk5kkw3jERWh
…ssertion dns.py _deploy_coredns() was reading `runtime_state.dns_output` to get the CoreDNS listen IP, but dns_output is None at deploy time (it's set after deployment completes). Fix: read from `context.spec.dns.root.listen_ip` which is always available. test_e2e_fullstack: substrate_output has no top-level 'healthy' key in the real handler (only nested per-subsystem keys). Replace the KeyError- prone assertion with `assert "networks" in substrate_output`. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NdYtbgMhKYfk5kkw3jERWh
Docker v1.48 rejects net.connect() on a container that was started with network_mode="none" (private mode): "container cannot be connected to multiple networks with one of the networks in private (none) mode". Replace the two-step (run with network_mode=none, then net.connect) with a single low-level API call that attaches the container to the core network with the static IP at creation time using create_networking_config / create_endpoint_config. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NdYtbgMhKYfk5kkw3jERWh
… on failure - Increase startup sleep from 2s to 3s to give CoreDNS more time - After container start, wait 1s and check it hasn't already exited - In _verify_dns_service, retry SOA query up to 6 times with 2s delays between attempts (12s total retry window) to handle slow-start scenarios - On all-retries-exhausted, capture and log CoreDNS container status and last 80 lines of logs for CI diagnosis
Without $TTL 3600 at the top of each zone file, miekg/dns (CoreDNS's zone parser) fails to anchor the SOA record to the zone origin and reports 'has no SOA record for origin root.internal.' causing CoreDNS to crash-loop. Add $TTL 3600 and explicit TTL+IN fields to every record in all three zone file generators.
apply_internet_policy and apply_peer_routing both called copy_to_container on the netengine_gateway container, which may not exist when running the gateway portal handler independently (e.g., in E2E federation tests that only provision substrate + DNS). docker.errors.NotFound was not being wrapped as GatewayError, so it propagated uncaught through _setup_peer. - apply_internet_policy: convert container NotFound to GatewayError - apply_peer_routing: same wrapping, preserving temp file cleanup - _apply_internet_policy: catch GatewayError and log a warning instead of aborting the whole handler — DNS forwarding can still be set up even without a gateway container Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NdYtbgMhKYfk5kkw3jERWh
CoreDNS container (coredns/coredns:1.11.3) does not have sh in its PATH, so _configure_peer_dns and _configure_upstream_resolver were failing with 'exec: sh: executable file not found in $PATH' when trying to append forwarding stubs via 'sh -c echo ... >> /etc/coredns/Corefile'. Replace the shell exec approach with a direct write to the host-mounted Corefile (context.zone_dir/Corefile), then send kill -HUP 1 to trigger CoreDNS config reload. This works because zone_dir is bind-mounted as /etc/coredns/ in the container, so host writes are immediately visible. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NdYtbgMhKYfk5kkw3jERWh
…and file reads
CoreDNS 1.11.3 runs from a scratch image with no shell utilities in PATH,
so exec+kill and exec+cat both fail. Replace them with daemon-level APIs:
- DockerHandler.signal_container(): new method using container.kill(signal=)
which sends signals via Docker daemon without needing a kill binary
- gateway_portal_handler: switch both Corefile reload calls from
exec_command(["kill", "-HUP", "1"]) to signal_container("HUP")
- test_e2e_federation: replace _read_corefile_from_container from
exec_run+cat to container.get_archive+tarfile extraction, which reads
files from any container regardless of available utilities
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NdYtbgMhKYfk5kkw3jERWh
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR adds two comprehensive end-to-end (E2E) integration test suites that validate the full NetEngine stack using real Docker containers, live DNS queries, ACME directory endpoints, and optional OIDC integration. These tests complement the existing unit and integration test suites by exercising real infrastructure components.
Key Changes
New Test Files
tests/integration/test_e2e_fullstack.py(297 lines)coreandplatformnetworks)root.internalNETENGINE_KEYCLOAK_URLenv var)tests/integration/test_e2e_federation.py(235 lines)Test Infrastructure Updates
tests/conftest.py: Added--run-e2epytest option to gate E2E tests (skipped by default)pytest_addoption()andpytest_collection_modifyitems()hooks.github/workflows/ci.yaml: Newe2ejob in CI pipeline--run-e2e -m "e2e and not slow"Source Code Modifications
netengine/handlers/dns.py: Changed CoreDNS volume mount from read-only (ro) to read-write (rw)netengine/handlers/docker_handler.py: Fixedexec_run()demux parameter handlingpyproject.toml: Updated pytest markerse2emarker for end-to-end testsslowmarker description to clarify scopeREADME.md: Updated roadmapNotable Implementation Details
Raw DNS Query Implementation: Tests use hand-crafted DNS SOA queries (no third-party DNS library) to validate CoreDNS responsiveness, keeping the test self-contained.
Docker Cleanup: Both test files include robust cleanup logic that removes all
netengine_*containers andcore/platformnetworks, even if tests fail.SSL/TLS Handling: ACME directory test disables certificate verification for self-signed step-ca certificates using
ssl.create_default_context()withCERT_NONE.Optional OIDC Test: Keycloak test gracefully skips if
NETENGINE_KEYCLOAK_URLis not set, allowing the test suite to run in CI without external dependencies.Async/Await Pattern: All tests use
@pytest.mark.asyncioand properly await handler execution, maintaining consistency with the async handler architecture.Marker-Based Filtering: Slow tests (step-ca image pull) are marked
@pytest.mark.slowand excludedhttps://claude.ai/code/session_01NdYtbgMhKYfk5kkw3jERWh