[BBT#565] Implement the BRAVO biased reader/writer lock

[devreal]

The goal of the BRAVO biased rwlock is to avoid readers taking the reader lock and thus contending for the atomic variable. Instead, readers raise a flag in an array to signal that they "took the lock" to any future writer. A writer takes the underlying atomic lock and waits for all readers to complete. While a writer has the lock, readers wait for the atomic rwlock.

The hash table in PaRSEC is a prime example for a use-case: writers are extremely rare since the resizing happens rarely and the max size is capped. However, every thread locking a bucket also takes a reader-lock. We can thus avoid the contention on the global lock for most of the application run.

The original proposal used a global hash table for all locks (https://arxiv.org/abs/1810.01553) but we use one array per lock. We know the number of threads in PaRSEC and can use fixed offsets, with padding to prevent cache line sharing (64B per thread). If an unknown thread takes the lock it goes straight to the atomic rwlock (unfortuntely, this includes the MPI progress thread at the moment).

This is still WIP, needs more testing and evaluation.

Open question:

• Does the comm thread not have an execution context? If so, can we access the global context in another way to know the number of threads in PaRSEC during initialization?

Original PR on Bitbucket: https://bitbucket.org/icldistcomp/parsec/pull-requests/565

Signed-off-by: Joseph Schuchart schuchart@icl.utk.edu