Regression in main: publish_metadata_and_ready drops tensors field, server stores 0 tensors per worker

[OrZipori]

Regression in main: publish_metadata_and_ready drops tensors field, server stores 0 tensors per worker

What happened

After deploying a vLLM source worker with --load-format modelexpress against a Redis-backed MX server, the server was logging:

Published metadata for 'MyModel' (source_id=..., worker_id=...): rank 0 (0 tensors)
Published metadata for 'MyModel' (source_id=..., worker_id=...): rank 1 (0 tensors)

...while the worker itself was logging right before that:

[Worker 0] Publishing 2585 tensors for model 'MyModel'

All target workers then fell back to disk loading because rdma_strategy.py skips any worker with no tensors and no worker_grpc_endpoint:

if not worker_tensor_descriptors(worker) and not worker.worker_grpc_endpoint:
    continue

Root cause

This is a regression introduced in commit 545bdf8 ("feat: add source-typed artifact metadata payloads #411"). That commit migrated WorkerMetadata construction in two places from the legacy tensors repeated field (field 3) to the new tensor_source oneof field (field 20), but it just swapped one for the other instead of writing both.

modelexpress/metadata/publish.py (non-P2P branch, the else in publish_metadata_and_ready):

# before 545bdf8 (v0.4.0 tag, works fine):
worker = p2p_pb2.WorkerMetadata(
    worker_rank=worker_rank,
    nixl_metadata=nixl_manager.nixl_metadata,
    tensors=tensor_protos,
)

# after 545bdf8 (current main, broken):
worker = p2p_pb2.WorkerMetadata(
    worker_rank=worker_rank,
    nixl_metadata=nixl_manager.nixl_metadata,
    tensor_source=tensor_source_metadata(tensor_protos),  # tensors= dropped
)

Same thing in modelexpress/trtllm_live_transfer.py (both publish_model_params and publish_from_worker).

The problem is the Redis backend log line in redis.rs reads directly from the legacy tensors field on the stored WorkerRecord:

info!("... rank {} ({} tensors)", worker_record.worker_rank, worker_record.tensors.len());

And WorkerRecord::from(WorkerMetadata) in backend.rs only populates tensors from source_payload.tensor_source — when the client sends tensor_source but not tensors, older server builds (e.g. the 0.4.0 container image) see tensors=[] because source_payload is an unknown proto3 field to them and gets silently discarded.

Even on fully updated server builds, the backend.rs round-trip (WorkerRecord → WorkerMetadata) already dual-writes both legacy_tensors and tensor_source. The client should do the same.

Affected scope

• v0.4.0 tag: NOT affected. tensors=tensor_protos was there and correct.
• main after 545bdf8: affected in publish.py and trtllm_live_transfer.py (3 call sites total).
• Only triggered on the non-P2P path (i.e. MX_P2P_METADATA not set and backend doesn't force it). The k8s-service backend is unaffected because REQUIRES_P2P_METADATA=True always takes the P2P branch, where tensor descriptors are served on-demand via GetTensorManifest and tensors= is never the delivery mechanism.

Fix

Just write both fields, same pattern the server already uses on the round-trip:

modelexpress/metadata/publish.py:

     worker = p2p_pb2.WorkerMetadata(
         worker_rank=worker_rank,
         nixl_metadata=nixl_manager.nixl_metadata,
+        tensors=tensor_protos,
         tensor_source=tensor_source_metadata(tensor_protos),
     )

modelexpress/trtllm_live_transfer.py (two call sites, same diff):

     worker = p2p_pb2.WorkerMetadata(
         worker_rank=mpi_rank,
         nixl_metadata=nixl_mgr.nixl_metadata,
+        tensors=tensor_protos,
         tensor_source=tensor_source_metadata(tensor_protos),
     )

Test gap

The existing test_calls_publish_and_starts_heartbeat in test_vllm_loader.py checks that mx_client.publish_metadata is called with the right identity and worker_id, but never inspects the WorkerMetadata proto itself. So the regression shipped silently. A test like this would have caught it:

def test_non_p2p_worker_metadata_includes_both_tensors_fields():
    # ... setup ...
    worker_proto = mx_client.publish_metadata.call_args.args[1]

    # legacy field — old servers only read from here
    assert len(worker_proto.tensors) == 3

    # new field — forward compat
    assert len(worker_proto.tensor_source.tensors) == 3

Environment

• Nebius cluster, B200 GPUs, TP=4
• vLLM with VLLM_PLUGINS=modelexpress, --load-format modelexpress
• Redis metadata backend (MX_METADATA_BACKEND=redis, no MX_P2P_METADATA)
• Client built from main (post-545bdf8), server image modelexpress-server:0.4.0
• Model: Kimi-K2.5-NVFP4, 2585 tensors per rank
• MX server logged rank N (0 tensors) for all source workers; all targets fell back to disk

Workaround

Either build the client from the v0.4.0 tag (which has the correct code), or patch publish.py at container startup to add tensors=tensor_protos, back.

[tanushriya910]

Fix merged in PR #504