`dist.broadcast` called unconditionally in `create_model_and_carry` breaks single-GPU runs

[dolceo]

Summary

pretrain.py is documented to support both torchrun and direct invocation (see launch() at pretrain.py:316, where WORLD_SIZE=1, RANK=0 are set and dist.init_process_group is only called if LOCAL_RANK in os.environ). However create_model_and_carry unconditionally calls dist.broadcast on every buffer:

# pretrain.py:147-148
for buffer in model.buffers():
    dist.broadcast(buffer, src=0)

When invoked without torchrun (single GPU), dist has never been initialized, and this raises RuntimeError: Default process group has not been initialized.

Root Cause

The buffer broadcast was added to keep RoPE cos/sin caches and zL_init buffers consistent across ranks, but no guard checks whether distributed mode is actually active.

Suggested Fix

if dist.is_initialized():
    for buffer in model.buffers():
        dist.broadcast(buffer, src=0)

The same fix is needed at apply_fsdp (fully_shard requires a process group), so single-GPU runs may need a separate non-FSDP code path. At minimum the README should be updated to state that torchrun is mandatory even for single-GPU.