Sorting intermediate data

[wonbaekimys]

This is the structure of intermediate data at reducer side.

Intermediate data
= (idata of key range 1, idata of key range 2, ..., idata of key range N)

idata of key range k
= (iblock 1, iblock 2, ..., iblock M)

Currently, for any idata of a specific key range, key and values are sorted within each iblock, but not across the entire iblocks.
As a result, a reducer will get a key with "partially sorted" list of values.

• It is required to sort values across the iblocks externally (the original eclipseMR is also missing this feature).

[vicentebolea]

I understand this issue. I wonder do we need the key values idata to be sorted externally to perform reduce operations. In my understanding the keys must be sorted only within the shuffle phase so that every node gets its corresponding key-values clusters. I might be wrong :)

What would happen if we do not sort those keys externally ?