IWriter performance evaluation

[wonbaekimys]

I evaluated of the performance of IWriter itself without network parts.

Test Method

This test mimics the procedure that messages::KeyValueShuffle instances arrives at TaskExecutor::write_key_value function and the arrived key-values are written into the object of IWriter class as follows:

1. Generate 256 MB of key and values. The generation solely takes around 30 seconds.
2. On every key-value generation, call IWriter::add_key_value function with the generated key-value.

We've been tested VeloxMR with kmeans using 10 GB data on 40 nodes, where the amount of shuffled data on each node is approximately 256 MB, so this test is performed with 256 MB key-values.

Parameters

• fine-lock (enable/disable): Fine-grained locking (Iwriter lock #45).
• std::move (enable/disable): To avoid data duplication when we insert key-values to iblock.
• iblock_size: The experiments are done with 4 MB, 8 MB, 16 MB, 32 MB and 64 MB of iblock size.

Result

I ran each test 8 times and take the average.

• IWriter::add_key_value function is called over 10 million times, and it takes no more than 2 minutes in total (regarding the key-value generation takes around 30 seconds).
• At least up to 64 MB, the performance improves as increasing iblock_size.
• As iblock_size increases, the performance gap among different optimization schemes (fine-lock and std::move) decreases.
• It seems that std::move rather degrades the performance.

[wonbaekimys]

It took 3min 34.57 sec when 40 threads share one IWriter object.

[wonbaekimys]

Introducing mutex on write_key_value function, it shows the better performance than a single threaded case.
256MB with iblock_size of 64 MB, it takes 75.79 seconds (around 95 seconds for single threaded case).
And without key-value generation time, it takes only 52.0933 seconds that IWriter object writes key-values onto disk.

I think IWriter itself is fast enough.

Here, std::move does not degrade the performance.**

[vicentebolea]

Great insights, I found that the main factor for our slow performance is the network contention.

I am trying to optimize it, yet I does not seem to be an easy task