What steps will reproduce the problem?
1. compile the following app with gcc a.c -lpthread
2. hpcrun ./a.out
#include <pthread.h>
#include <stdlib.h>
#include <stdio.h>
#define THREADS_BEFORE_MAIN 1
typedef struct {
int id;
pthread_t tid;
pthread_barrier_t * barrier;
long x;
void * ret;
} thread;
const int n_threads = 10;
const long x = 25;
long fib(long n) {
if (n < 2) {
return 1;
} else {
return fib(n - 1) + fib(n - 2);
}
}
void * f (void * t_) {
thread * t = t_;
printf("thread %d\n", t->id);
pthread_barrier_wait(t->barrier);
t->ret = (void *)fib(t->x);
return t;
}
thread * T;
pthread_barrier_t * barrier;
#if THREADS_BEFORE_MAIN
__attribute__ ((constructor))
#endif
static int create_threads() {
int i;
barrier = malloc(sizeof(pthread_barrier_t));
pthread_barrier_init(barrier, 0, n_threads);
T = malloc(sizeof(thread) * n_threads);
for (i = 1; i < n_threads; i++) {
T[i].id = i;
T[i].x = x;
T[i].barrier = barrier;
pthread_create(&T[i].tid, 0, f, &T[i]);
}
T[0].id = 0;
T[0].x = x;
T[0].barrier = barrier;
T[0].tid = pthread_self();
f(&T[0]);
return 1;
}
int join_threads() {
int i;
for (i = 1; i < n_threads; i++) {
pthread_join(T[i].tid, 0);
}
return 1;
}
int main(int argc, char ** argv) {
int i;
#if !THREADS_BEFORE_MAIN
create_threads();
#endif
join_threads();
for (i = 0; i < n_threads; i++) {
printf("ans[%ld] = %ld\n", i, (long)T[i].ret);
}
return 0;
}
What is the expected output? What do you see instead?
It should finish; Each thread prints its thread number; results of all threads
should also be printed, like this.
andromeda:p% hpcrun ./a.out
thread 1
thread 3
thread 5
thread 4
thread 2
thread 7
thread 6
thread 8
thread 0
thread 9
ans[0] = 121393
ans[1] = 121393
ans[2] = 121393
ans[3] = 121393
ans[4] = 121393
ans[5] = 121393
ans[6] = 121393
ans[7] = 121393
ans[8] = 121393
ans[9] = 121393
When run, it actually deadlocks.
Please use labels and text to provide additional information.
One day, I encounter a problem that hpctoolkit's hpctraceviewer does not work
together with MassiveThreads. In short, hpctraceviewer displays the result as
if the app is single-threaded, no matter how many threads you create. It seems
that it is caused due to the following.
* (guess) hpctoolkit wants to hook calls to pthread_create
* MassiveThreads does not directly call pthread_create; instead, it obtains a function pointer to pthread_create by dlopen and dlsym and calls it through the function pointer
* (guess) hpctoolkit fails to hook pthread_create called this way
This is simply a guess. Regardless whether this guess is collect, it seems
logical to try to call pthread_create and all related functions directly so as
to make MassiveThreads more normal from hpctoolkit's point of view.
When I changed the source code of MassiveThreads this way (i.e., replace
real_pthread_create in myth_init.c to pthread_create), hpcrun deadlocked; the
application does not reach its main.
I then found that this is caused due to the following interplay between
MassiveThreads an hpctoolkit.
*# MassiveThreads creates worker threads before entering main.
*# To do so, MassiveThreads performs the following loop (myth_init.c). Each worker calls myth_worker_thread_fn. Note that it creates only (num_threads - 1) threads, as the app already has one (main) thread. After creating threads, it calls myth_worker_thread_fn function directly.
{{{
for(i = 1; i < num_threads; i++) {
pthread_create(..., myth_worker_thread_fn, ...);
}
myth_worker_thread_fn(...);
}}}
*# myth_worker_thread_fn performs a barrier (pthread_barrier_wait).
gdb told us that the main thread is stuck inside pthread_barrier_wait, whereas
all other threads are inside nanosleep. To reproduce the problem, I made the
following simple app that has nothing to do with MassiveThreads. This
application deadlocks when run with hpcrun.
{{{
gcc -g -O0 a.c -lpthread
hpcrun -t ./a.out
}}}
All of the above three conditions seem important.
* If we remove barrier, it works fine
* If we instead create n threads it works fine
* If we instead create (n-1) threads in the main function, it works fine
My hypothesis of the problem's root cause is:
* hpctoolkit somehow does not want threads to be started before the application enters the main, so all threads created before entering the main are suspended until the main thread reaches main
Regardless whether this is true or not, there are several workarounds that we
can implement in MassiveThreads and that are useful anyways.
* avoid using __attribute__((constructor)) for initializing threads. instead check if the library is already initialized before each function directly called by the client program. this way we can still free the programmer from explicitly initializing the library
Original issue reported on code.google.com by
Taura.Kenjiro@gmail.comon 4 Aug 2014 at 6:23