Merlin LinkControl initialization error in simple 2-router mesh (TrafficGen endpoints)

[gandhiomkar]

Hi ,
I’m new to SST framework and trying to build a very simple setup: a peer-to-peer network with two routers directly connected, each with one merlin.trafficgen endpoint. I’ve run into a LinkControl initialization error

Setup

• SST version: 14.1.0 (also reproduced on 15.x)
• Topology: 2 routers using merlin.mesh (shape 2:1)
• Each router has 1 local port connected to a trafficgen endpoint
• Routers are directly connected to each other

here is the network graph created by sst with graphViz:

Error

I consistently get the following fatal error during initialization:

Error during LinkControl protocol initialization. The most likely cause of this is connecting an endpoint to a router port expecting to be connected to another router.

full stack trace:
error:

FATAL: Rank: 0,0, time: 0 - called in file: interfaces/linkControl.cc, line: 224, function: init
Element name: endpoint1,  type: merlin.linkcontrol (full type tree: merlin.trafficgen.merlin.linkcontrol)
Error during LinkControl protocol initialization.  The most likely cause of this is connecting an endpoint to a router port expecting to be connnected to another router.

SST Fatal Backtrace Information:
    0 : sst(_ZNK3SST6Output5fatalEjPKcS2_iS2_z+0x38a) [0x5ea0085400a6]
    1 : sst(_ZNK3SST13BaseComponent6vfatalEjPKcS2_iS2_P13__va_list_tag+0x324) [0x5ea008461992]
    2 : sst(_ZNK3SST13BaseComponent10sst_assertEbjPKcS2_iS2_z+0xf2) [0x5ea008461d3a]
    3 : /home/admin/sst-14-install/lib/sst-elements-library/libmerlin.so(_ZN3SST6Merlin11LinkControl17checkInitProtocolEPNS_5EventENS0_12RtrInitEvent8CommandsEjPKcS7_+0xaa) [0x7873b3e8a92a]
    4 : /home/admin/sst-14-install/lib/sst-elements-library/libmerlin.so(_ZN3SST6Merlin11LinkControl4initEj+0x249) [0x7873b3e8ab7d]
    5 : /home/admin/sst-14-install/lib/sst-elements-library/libmerlin.so(_ZN3SST6Merlin10TrafficGen4initEj+0x3d) [0x7873b3e74097]
    6 : sst(_ZN3SST15Simulation_impl10initializeEv+0xf1) [0x5ea0085791df]
    7 : sst(+0x5390c8) [0x5ea0084060c8]
    8 : sst(main+0x2237) [0x5ea008408943]
    9 : /lib/x86_64-linux-gnu/libc.so.6(+0x2a1ca) [0x7873bbe2a1ca]
   10 : /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x8b) [0x7873bbe2a28b]
   11 : sst(_start+0x25) [0x5ea008403bb5]
--------------------------------------------------------------------------
MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD
with errorcode 1.

Debugging details

I traced the issue into LinkControl::checkInitProtocol():

• The code expects a RtrInitEvent::REPORT_FLIT_SIZE
• However, the received event has command REPORT_ID
• This causes the assertion to fail

Relevant snippet:

ev = rtr_link->recvUntimedData();
init_ev = checkInitProtocol(ev, RtrInitEvent::REPORT_FLIT_SIZE, CALL_INFO);

RtrInitEvent* LinkControl::checkInitProtocol(Event* ev, RtrInitEvent::Commands command, uint32_t line, const char* file, const char* func)
{
    bool good = true;
    RtrInitEvent* init_ev = nullptr;
    // Check to make sure the event isn't null and that it is an init event
    if ( nullptr == ev || static_cast<BaseRtrEvent*>(ev)->getType() != BaseRtrEvent::INITIALIZATION ) good = false;

    if ( good ) {
        init_ev = static_cast<RtrInitEvent*>(ev);

        // Now check to make sure this is the right protocol event
        if ( init_ev->command != command ) {
            good = false;
        }
    }

    sst_assert(good, line, file, func, 1, "Error during LinkControl protocol initialization.  The most likely cause of this is connecting an endpoint to a router port expecting to be connnected to another router.\n");
    return init_ev;
}

From what I understand, this suggests a mismatch in initialization protocol between components.

My current configuration

import sst

link_bw = "5GB/s"
flit_size = "72B"
lat = "1ns"
buf_size = "1440B"

router_params = {
    "link_bw": link_bw,
    "flit_size": flit_size,
    "xbar_bw": "5GB/s",
    "input_buf_size": buf_size,
    "output_buf_size": buf_size,
    "local_ports": 1,
    "num_ports": 3,
    "input_latency": "1ns",
    "output_latency": "1ns",
    "vn_remap": "linear",
    "topology": "topology"
}

topo_params = {
    "mesh.shape": "2:1",
    "mesh.width": "1:1",
    "mesh.local_ports": 1,
    "network_name": "main_net"
}

routers = {}
for i in range(2):
    rtr = sst.Component(f"router{i}", "merlin.hr_router")
    rtr.addParams(router_params)
    rtr.addParam("id", i)

    topo = rtr.setSubComponent("topology", "merlin.mesh")
    topo.addParams(topo_params)

    ep = sst.Component(f"endpoint{i}", "merlin.trafficgen")
    ep.addParams({
        "id": i,
        "num_peers": 2,
        "num_vns": 1,
        "link_bw": "5GB/s",
        "topology": "topology",
        "packet_size": "256B",
        "PacketDest.pattern": "Uniform",
        "PacketSize.pattern": "Uniform",
        "PacketDelay.pattern": "Uniform"
    })

    ep_topo = ep.setSubComponent("topology", "merlin.mesh")
    ep_topo.addParams(topo_params)

    link_host = sst.Link(f"link_host{i}")
    link_host.connect((ep, "rtr", lat), (rtr, "port0", lat))

    routers[i] = rtr

link_inter = sst.Link("inter_router_link")
link_inter.connect(
    (routers[0], "port1", lat),
    (routers[1], "port1", lat)
)

link_inter_back = sst.Link("inter_router_link_back")
link_inter_back.connect(
    (routers[1], "port2", lat),
    (routers[0], "port2", lat)
)

What I’ve tried

• Verified parameters using sst-info
• Tested on both SST 15.x (latest master branch from GitHub) and v14.1.0
• Stepped through initialization to confirm the protocol mismatch (REPORT_ID vs REPORT_FLIT_SIZE)
• Tried enabling/disabling explicit linkcontrol subcomponent (commented section in code)

Questions

1. Is my component wiring correct?
   Specifically, is it valid to directly connect trafficgen → router port like this, or is an explicit merlin.linkcontrol subcomponent required?

2. Are the router ports configured correctly?
   I’m unsure if ports 1 and 2 are being interpreted as router-to-router links correctly in this setup.

3. Does this error indicate a topology mismatch?
   Could the merlin.mesh topology be expecting different connectivity than what I’ve manually wired?

4. Is there any required initialization ordering or synchronization
   for LinkControl that I might be missing, or is this purely a configuration issue?

5. Is there a minimal working example
   of a 2-node Merlin network with trafficgen endpoints that I can compare against?

---

I’d really appreciate any guidance on whether this is a configuration issue or a misunderstanding of how Merlin components should be connected. I’ve tried to debug the initialization path before posting, but I may be missing something fundamental.

Thanks in advance for your help!

[feldergast]

We're working on the SST 16 release, so I don't have time to dig into this fully, but you are likely connecting the ports wrong. The topology objects expect each port to be used for a specific type of connection. Connecting everything correctly for a merlin simulation can be tricky, so the merlin library provides several python modules that can be used to create all the connections. There is a working trafficgen simulation using the torus topology in merlin/tests/torus_5_trafficgen.py. However, this uses the older merlin python module, which will soon be removed and we haven't created the python helpers for trafficgen in the newer python modules. For an example of how the new python modules work, you can look in merlin/test/dragonfly_128_test.py (among others). After the release goes out, I can add trafficgen to the new infrastructure. Earliest I can get to that is likely next week.

[deanchester]

Diagnosis

This is a configuration issue, not a Merlin bug. There are three independent problems in the script — but the one that produces the LinkControl protocol initialization error is the wiring of the endpoint to the wrong router port.
I confirmed this by reproducing the failure against current sst-elements master in an isolated SST session, and the corrected config at the bottom runs to completion (Simulation is complete, simulated time: 492 ns).

1. mesh.shape / mesh.width use x as the separator, not :

merlin/topology/mesh.cc parses the shape like this:

// mesh.cc:41
dimensions = std::count(shape.begin(), shape.end(), 'x') + 1;
// mesh.cc:253-264 parseDimString:
end = shape.find('x', start);
... strtol(sub.c_str(), NULL, 0);

So "2:1" contains zero 'x' characters → dimensions = 1, and strtol("2:1", ...) stops at : and returns 2. You silently get a 1-D mesh of size 2, not a 2-D 2x1 mesh as the parameter name suggests. (Same for mesh.width="1:1" → width 1.) That much actually happens to work, but it's worth knowing — for a real 2D mesh you must write e.g. "4x4". For a 1D mesh of 2, just use "2".

2. Port numbering — the endpoint is connected to a router-to-router port

For a 1-D mesh with dim_size=2, dim_width=1, local_ports=1, mesh.cc lays the ports out as:

// mesh.cc:57-77
port_start[0][0] = 0;     // +X router-to-router
port_start[0][1] = 1;     // -X router-to-router
local_port_start = needed_ports = 2 * dim_width = 2;   // local/host ports start here

And getPortState() (mesh.cc:207–234) returns:

router	port 0 (+X)	port 1 (−X)	port 2 (local)
id=0	R2R → must connect to rtr1	UNCONNECTED (id_loc==0)	R2N (host)
id=1	UNCONNECTED (id_loc==dim_size−1)	R2R → connects back to rtr0	R2N (host)
Your script does:

link_host.connect((ep, "rtr", lat), (rtr, "port0", lat))     # endpoint -> R2R port  ❌
link_inter.connect((rtr0, "port1"), (rtr1, "port1"))         # rtr0.port1 is UNCONNECTED ❌
link_inter_back.connect((rtr1, "port2"), (rtr0, "port2"))    # port2 is the LOCAL host port ❌

That's why you hit the assert. Look at merlin/interfaces/portControl.cc line 622–660:

if ( topo->isHostPort(port_number) ) {                       // host port path
    /* sends REPORT_BW, REPORT_FLIT_SIZE, REPORT_ID */
} else {
    /* router-to-router path: sends REPORT_BW, REPORT_ID, REPORT_PORT */
}

Because endpoint1 was attached to port0 of router1, the topology says port0 is R2R (it's the +X port). PortControl therefore takes the else branch and after REPORT_BW sends REPORT_ID instead of REPORT_FLIT_SIZE. The endpoint's LinkControl::init() (linkControl.cc:226–227) calls checkInitProtocol(ev, REPORT_FLIT_SIZE, ...) and asserts. That is exactly the REPORT_ID vs REPORT_FLIT_SIZE mismatch you observed.

3. The merlin.trafficgen endpoint does not have a topology slot

// merlin/trafficgen/trafficgen.h:105
SST_ELI_DOCUMENT_SUBCOMPONENT_SLOTS(
    {"networkIF", "Network interface", "SST::Interfaces::SimpleNetwork" }
)

So ep.setSubComponent("topology", "merlin.mesh") on the endpoint is silently ignored. The topology subcomponent only belongs on the router. Your endpoint params should also drop "topology": "topology". Trafficgen auto-loads merlin.linkcontrol for its networkIF slot if you don't set one explicitly.

Note on the second inter-router link

In a non-torus 1D mesh of size 2 there is only one inter-router link. rtr0.port1 (-X) and rtr1.port0 (+X) are both legitimately UNCONNECTED. If you want two parallel links between the routers, increase mesh.width (e.g. "2" for two parallel links per dim) or use merlin.torus.

---

Working configuration

import sst
link_bw  = "5GB/s"
flit_size = "72B"
lat = "1ns"
buf_size = "1440B"
router_params = {
    "link_bw": link_bw,
    "flit_size": flit_size,
    "xbar_bw": "5GB/s",
    "input_buf_size": buf_size,
    "output_buf_size": buf_size,
    "num_ports": 3,            # 2 R2R + 1 local
    "input_latency": "1ns",
    "output_latency": "1ns",
    "topology": "topology",
}
# 1-D mesh of size 2; 'x' separates dims (so just "2" for 1-D).
topo_params = {
    "shape": "2",
    "width": "1",
    "local_ports": 1,
}
routers = {}
for i in range(2):
    rtr = sst.Component(f"router{i}", "merlin.hr_router")
    rtr.addParams(router_params)
    rtr.addParam("id", i)
    rtr.setSubComponent("topology", "merlin.mesh").addParams(topo_params)
    routers[i] = rtr
    ep = sst.Component(f"endpoint{i}", "merlin.trafficgen")
    ep.addParams({
        "id": i,
        "num_peers": 2,
        "num_vns": 1,
        "link_bw": link_bw,
        "packets_to_send": 50,
        "packet_size": "64B",
        "message_rate": "1GHz",
        "PacketDest.pattern": "Uniform",
        "PacketDest.RangeMin": 0,
        "PacketDest.RangeMax": 2,
        "PacketSize.pattern": "Uniform",
        "PacketSize.RangeMin": "16B",
        "PacketSize.RangeMax": "64B",
        "PacketDelay.pattern": "Uniform",
        "PacketDelay.RangeMin": "1ns",
        "PacketDelay.RangeMax": "5ns",
    })
    # Endpoint connects to the LOCAL host port, which for this config is port 2
    # (ports 0 and 1 are reserved for +X / -X router-to-router).
    sst.Link(f"link_host{i}").connect((ep, "rtr", lat), (rtr, "port2", lat))

# Single inter-router link: rtr0 +X (port 0) <-> rtr1 -X (port 1).
sst.Link("inter_router_link").connect(
    (routers[0], "port0", lat),
    (routers[1], "port1", lat),
)

Verified on current sst-elements master against sst-core master:

Simulation is complete, simulated time: 492 ns

Quick answers to your numbered questions

1. Wiring: trafficgen → router directly is fine; merlin.trafficgen auto-loads merlin.linkcontrol when you don't set the networkIF slot. The bug was attaching to a router-to-router port, not the missing linkcontrol.
2. Router ports: for mesh shape=N width=W local_ports=L, ports are laid out as [+X0..+X(W-1), -X0..-X(W-1), +Y..., -Y..., ..., local_0 .. local_(L-1)]. Local ports always come last. Use port{local_port_start + k} for the k-th host.
3. Topology mismatch: yes — your wiring to port0/port1/port2 did not match what merlin.mesh expected those ports to be (R2R vs R2N).
4. Init ordering: nothing on your side is needed; LinkControl::init and PortControl::init handshake automatically — you just need both ends to agree on whether the link is host-side or router-side, which is determined by the topology's getPortState(port_number) / isHostPort(port_number).
5. Minimal working example: src/sst/elements/merlin/tests/torus_5_trafficgen.py is the canonical reference (5-node torus + trafficgen via the high-level pymerlin.py helper). The script above is the same idea trimmed to two routers and built using the low-level API like yours.

Hope this helps — closing thought: a more user-friendly behavior here would be for LinkControl/PortControl to detect a router-side peer and report the offending router id and port number in the fatal message.