Skip to content

eMuleAI network parity: interest, and the libutp vendoring question #1177

Description

@kno

Summary

I maintain a fork at kno/amule that implements the
wire-level extensions an eMuleAI peer expects, so that aMule and eMuleAI
recognise each other's capabilities instead of falling back to the plain eD2k
subset. It is built on 3.0.1 and tracks this repository.

The code for everything described here, including the rendezvous framing
discussed in this thread, lives on the
integration/network-parity
branch. The fork's master mirrors this repository plus downstream tooling
only, so a branch cut from it starts clean.

Before preparing any pull request I would rather ask whether you want this at
all, and under what conditions — particularly the one in "Open question" below,
which decides whether two of the pieces are even offerable.

What it consists of

Seven pieces, each with its own tests, in dependency order:

# Piece Depends on
1 Peer capability recognition — the CT_MOD_MISCOPTIONS tag and its bits, plus demultiplexing of OP_UDPRESERVEDPROT2 frame types
2 Kademlia protocol 0x0a — version advertisement and the AICH gate
3 Address widening — a family-agnostic address type, so an IPv6 peer is not a peer with "no address"
4 IPv6 dual stack — one listening socket per family; reachability advertised only after inbound IPv6 is observed 3
5 uTP transport — sharing the eD2k UDP port 1, 3
6 NAT traversal — answering a relayed rendezvous, rate-limited, with bounded hole punches 1, 5
7 QUIC NAT-T transport 6

1 and 2 are small and self-contained. 3 is the one that unblocks the rest and
also the most invasive of the early ones.

Why the formats can be trusted

These are not guesses at eMuleAI's wire format. They were measured against a
running eMuleAI v1.6.0:

  • CT_MOD_MISCOPTIONS reads 0x0000001F from it — every bit of the mask this
    work defines, matching bit for bit.
  • The EAQN1 proof magic and the ALPN string ed2k-ai-natt-quic-v1 both appear
    verbatim in its binary.
  • Its GnuTLS priority string offers the same four TLS 1.3 ciphers in the same
    order, over a subset of the same groups.
  • An eD2k handshake over native IPv6 completes between the two
    implementations — a hand-built OP_HELLO to its port answered with
    OP_HELLOANSWER.

Open question, and the reason for asking first

Piece 5 vendors libutp under src/extern/, as a byte-identical copy of
transmission/libutp at 490874c44a2e
(MIT), with no local patches and a provenance note. Pieces 5, 6 and 7 have no
meaning without a uTP implementation.

To be precise about what is and is not proposed: the bundled copy is the
default, not the only route. cmake/libutp.cmake offers both, following the
same bundled-or-system convention this project already uses for picojson
(USE_SYSTEM_PICOJSON):

  • USE_SYSTEM_LIBUTP=OFF (default) — the vendored copy.
  • USE_SYSTEM_LIBUTP=ON — a system-installed libutp; fails loudly if absent,
    rather than silently downgrading to a green build with the transport missing at
    runtime.

The default is the bundled copy because libutp is not packaged anywhere I can
find
. Measured rather than assumed:

  • Debian trixie: no libutp-dev, no libutp, and apt-cache search returns
    nothing; sources.debian.org reports no exact match.
  • Homebrew: no formula (brew search libutp offers only unrelated names).

So making the system route the default would break the build on most machines
with a FATAL_ERROR asking the user to install something that does not exist as
a package.

Vendoring is still a project-policy decision rather than an engineering one, which
is why I would rather hear your position than write a pull request that assumes
it. If it is unacceptable, uTP does not disappear — it becomes conditional on a
library your users would have to build by hand, and that changes what pieces 5–7
can honestly promise.

One property worth stating because it is what makes the copy maintainable: it is
byte-identical to the pinned upstream commit, with no local patches, and the
vendored CMakeLists.txt is not edited either — the build adapts to upstream's
target name instead. A patched third-party file turns every version bump into a
merge and makes the provenance note untrue.

What I would not offer yet

Being explicit so this is not discovered in review:

  • QUIC (piece 7) is not ready. It compiles, negotiates TLS, classifies frames
    and advertises its capability bit, but no QUIC connection has ever completed
    with another implementation
    , and one decision inside its authentication proof
    was made without evidence. I would not ask anyone to maintain that yet.
  • NAT traversal (piece 6) answers but does not initiate. Choosing a relay is
    deliberately gated off, because whether a candidate relay can reach a firewalled
    peer is not knowable from the requesting side. IPv6 hole punches encode and parse
    but are not sent.
  • Nothing in the fork's own packaging, container images or translation catalogs is
    proposed — those are downstream concerns and would conflict with your Weblate
    pipeline.

What I am asking

  1. Is this direction something you want in aMule at all?
  2. What is your position on vendoring libutp under src/extern/?
  3. If yes to (1): would you prefer one piece at a time in the order above,
    starting with capability recognition, or a different shape?

Happy to adapt to whatever review process suits you, including splitting
pieces further.

Test plan

Each piece carries unit tests. The current fork tip is at 73/73 in both build
configurations, with CI green across Ubuntu, macOS and mingw-w64 (Debug and
Release), plus clang-tidy and clang-format. Per-piece test counts and evidence
would come with each pull request.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions