Curious about correlation between publish rate and latency

[cvilas]

There seems to be an apparent correlation between publishing rate and snd/recv latency which I am unable to understand. I am running ecal_sample_latency_snd and ecal_sample_latency_rec on the same host, with the snd application slightly modified to accept a command line parameter for a configurable delay/interval between each publish. (I will post the diff in the thread). With increasing delays, I see increase in latency as follows

Publisher terminal

ecal_sample_latency_snd -s 1024 -d 0
--------------------------------------------
Runs                    : 5000
Message size            : 1024 kB
Memory buffer           : 1
Delay between sends     : 0 ms
Zero copy               : OFF
[eCAL][Config] Specified yaml configuration path not valid:"ecal.yaml". Using default configuration.
1771920156178 ms | vilas-t14s | latency_snd | 6444 | info | Using built-in local time plugin.
Messages sent           : 5000
--------------------------------------------

ecal_sample_latency_snd -s 1024 -d 1 
--------------------------------------------
Runs                    : 5000
Message size            : 1024 kB
Memory buffer           : 1
Delay between sends     : 1 ms
Zero copy               : OFF
[eCAL][Config] Specified yaml configuration path not valid:"ecal.yaml". Using default configuration.
1771920180020 ms | vilas-t14s | latency_snd | 6504 | info | Using built-in local time plugin.
Messages sent           : 5000
--------------------------------------------

ecal_sample_latency_snd -s 1024 -d 10
--------------------------------------------
Runs                    : 5000
Message size            : 1024 kB
Memory buffer           : 1
Delay between sends     : 10 ms
Zero copy               : OFF
[eCAL][Config] Specified yaml configuration path not valid:"ecal.yaml". Using default configuration.
1771920192931 ms | vilas-t14s | latency_snd | 6551 | info | Using built-in local time plugin.
Messages sent           : 5000

Outputs at the subscriber

# For 0ms delay between publish
--------------------------------------------
Messages received             : 5000
Message size received         : 1024 kB
Message average latency       : 84 us
Message min latency           : 77 us @ 256
Message max latency           : 405 us @ 2565
Throughput                    : 11863 MB/s
                              : 11 GB/s
                              : 11 kMsg/s
--------------------------------------------

# For 1ms delay between publish
--------------------------------------------
Messages received             : 5000
Message size received         : 1024 kB
Message average latency       : 207 us
Message min latency           : 142 us @ 2767
Message max latency           : 496 us @ 2365
Throughput                    : 4810 MB/s
                              : 4 GB/s
                              : 4 kMsg/s
--------------------------------------------

# For 10ms delay between publish
--------------------------------------------
Messages received             : 5000
Message size received         : 1024 kB
Message average latency       : 608 us
Message min latency           : 219 us @ 3760
Message max latency           : 1706 us @ 975
Throughput                    : 1642 MB/s
                              : 1 GB/s
                              : 1 kMsg/s
--------------------------------------------

Why should this be? Shouldn't the per-message latency be the same no matter what the inter-publish delay?

(This is on an X86-64 linux PC)

[cvilas]

diff.txt