Skip to content

Spread keyless records across ranges, or stop promising that #42

Description

@HectorIFC

Context

docs/guides/produce-and-consume.md:55-56 tells users:

Omit the key and records are distributed across ranges, which maximises parallelism and gives you no ordering between them.

The code does the opposite. Routing hashes the key with :erlang.phash2(key, keyspace_size) (lib/malachi/keyspace.ex:48) and nil is just another term:

iex> :erlang.phash2(nil, 256)
252

It is a constant. So every keyless record in a topic hashes to the same position, lands in the same range, and is serialized by that range's single primary. Instead of maximum parallelism, keyless traffic gets the narrowest possible path, and it is fully ordered rather than unordered.

The impact is a write hotspot exactly for the workload the documentation steers users toward when they do not need ordering, plus a promise the system does not keep.

Plan

Two honest ways out; the PR should pick one and say why.

  1. Make it true: route a record with no key to a range chosen per record (round-robin over the active ranges, or a random position in the keyspace). This delivers the documented parallelism and the documented absence of ordering. It touches Malachi.Broker's routing (owning_range_id/3 and the grouping around it) and must keep the single-range fast path correct.
  2. Make the docs true: state that keyless records share one range, are ordered among themselves, and do not scale out, and point users who want spread at supplying a key (even a random one). Zero risk, and it may be the right answer if the model prefers one rule for every record.

Whichever is chosen, docs/guides/produce-and-consume.md and the diagram above that paragraph must end up matching the behaviour.

Risks and open questions

  • Option 1 makes routing non-deterministic for keyless records, which is fine for ordering (there is none to preserve) but changes what a reader sees on a replay; the consume side already tolerates it, since a consumer walks ranges rather than keys.
  • Option 1 interacts with the split policy: keyless traffic would suddenly spread its load, which changes what a size or rate trigger observes.
  • Check whether any test or benchmark relies on keyless records landing together before changing the routing.

Verification

  • A test asserting the chosen behaviour explicitly (either records with no key land in more than one range, or they land in exactly one and the docs say so).
  • Full suite, mix credo --strict, mix dialyzer.
  • If option 1 is chosen, a loadtest with keyless records to confirm the hotspot is gone.

Metadata

Metadata

Assignees

Labels

bugSomething isn't workingdocumentationImprovements or additions to documentation

Projects

Status
Backlog

Relationships

None yet

Development

No branches or pull requests

Issue actions