Statically linked binaries

[mhoff]

We should provide statically linked binaries for users who just want to use our application as convenient as possible. @stefanorf, @briehm I think somebody already looked into this issue.
If yes, could you please provide your findings. If not, please also reply to this issue so that we can plan & assign this issue appropriately.

[briehm]

We first need to statically compile the MPI library, i.e.

./configure --without-memory-manager --without-libnuma --enable-static --disable-shared

http://www.open-mpi.de/faq/?category=mpi-apps#static-mpi-apps

After installing MPI and PLL, I tried to build a static version of our code via:

mpicc <Obj_files.o> -static -Wl,--start-group -lpll-sse3 -lmpi -Wl,--end-group -lm -lrt

Then I got tons of errors of the following kind:

<some_file.c:(.text+0x42)> undefined reference to `some_variable / function'
<some_lib.a:(.text+0x42)> In function `some_function': undefined reference to `some_variable / function'

This kind of problem often occurs, if the libraries are in the wrong order, but with the linking option --start-group/end-group this should not be the case.
http://stackoverflow.com/questions/45135/why-does-the-order-in-which-libraries-are-linked-sometimes-cause-errors-in-gcc

[mhoff]

First of all, isn't it enough to statically link PLL and dynamically link MPI? Is this even possible?
@stamatak what is your opinion regarding this issue?

https://www.open-mpi.org/faq/?category=mpi-apps#static-mpi-apps

Fully static linking is not for the weak, and it is not recommended. But it is possible, with some caveats.

Nevertheless I will now also try to statically link both libraries (as this is the most convenient way of providing our application). But I doubt, that this can be very efficient in relation to a dynamically linked MPI implementation in the OS.

[mhoff]

Hybrid linking (static PLL, shared MPI) should be possible using:
LFLAGS=-Wl,-Bstatic -lpll-sse3 -Wl,-Bdynamic $(shell mpicc --showme:link) -lm -lrt
The resulting binary has a size of 2.0M and the 'normal' one of 40K.
Also ldd shows no sign of PLL:

$ ldd pltb.out
linux-vdso.so.1 =>  (0x00007ffe94c23000)
libmpi.so.1 => /usr/lib/libmpi.so.1 (0x00007efec6272000)
libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007efec5f6c000)
librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007efec5d64000)
libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007efec5b46000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007efec5782000)
libutil.so.1 => /lib/x86_64-linux-gnu/libutil.so.1 (0x00007efec557f000)
libhwloc.so.5 => /usr/lib/x86_64-linux-gnu/libhwloc.so.5 (0x00007efec533f000)
libltdl.so.7 => /usr/lib/x86_64-linux-gnu/libltdl.so.7 (0x00007efec5135000)
/lib64/ld-linux-x86-64.so.2 (0x00007efec65f3000)
libnuma.so.1 => /usr/lib/x86_64-linux-gnu/libnuma.so.1 (0x00007efec4f2a000)
libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007efec4d26000)

@briehm Can you verify this on your Darwin-based system?

I am just now evaluating how to integrate this configuration with mpicc in a correct manner.
Calling mpicc like before does not sound right as we are then supplying two linking configurations to gcc. In fact it seems to work that way, but I don't have a good feeling about this.

On my system mpicc wraps the following call:
gcc -I/usr/lib/openmpi/include -I/usr/lib/openmpi/include/openmpi -pthread -L/usr//lib -L/usr/lib/openmpi/lib -lmpi -ldl -lhwloc
which is nearly equal to
mpicc --showme:command` `mpicc --showme:compile` `mpicc --showme:link
producing
gcc -I/usr/lib/openmpi/include -I/usr/lib/openmpi/include/openmpi -pthread -pthread -L/usr//lib -L/usr/lib/openmpi/lib -lmpi -ldl -lhwloc
Note the redundant -pthread.
Hence, replacing the call to mpicc with the above command, while substituting the "link" part with our custom configuration produces what we want with an extra -pthread.
Opinions on that?

[mhoff]

I pushed the mixed version (ff74183).

[stamatak]

thanks, in any case I guess a statically linked sequential and PThreads
based versionshould be sufficient for most users, I didn't think about
MPI issues

alexis

On 14.06.2015 20:14, Michael Hoff wrote:

I pushed the mixed version (ff74183
ff74183).

—
Reply to this email directly or view it on GitHub
#4 (comment).

Alexandros (Alexis) Stamatakis

Research Group Leader, Heidelberg Institute for Theoretical Studies
Full Professor, Dept. of Informatics, Karlsruhe Institute of Technology
Adjunct Professor, Dept. of Ecology and Evolutionary Biology, University
of Arizona at Tucson

www.exelixis-lab.org

[briehm]

One needs to alter the linker flags for OS X, because it does not use the GNU linker and thus does not have the Bstatic and Bdynamic options:
LFLAGS=-L/usr/local/lib/static -lpll-sse3 -Wl,-dynamic $(shell mpicc --showme:link) -lm
Prior one needs to move the static libraries into a dedicated folder, such that the linker does not use the dynamic libs instead (it looks for dynamic libs first, then static libs afterwards..)
Interestingly, the resulting binary is 706K in size.

[mhoff]

Did you verify the correctness of your process with ldd?
Is there no -Wl,-static variant for OS X?
Does the folder name imply the kind of linking?
This solution seems weird to me.

[briehm]

Yes, I checked this with the program otool (similar tool to ldd).

/opt/local/lib/mpich-mp/libmpi.12.dylib (compatibility version 13.0.0, current version 13.5.0)
/opt/local/lib/mpich-mp/libpmpi.12.dylib (compatibility version 13.0.0, current version 13.5.0)
/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1213.0.0)
/opt/local/lib/libgcc/libgcc_s.1.dylib (compatibility version 1.0.0, current version 1.0.0)

If someone wants to compile our program, he would need MPI and GCC installed. The PLL is linked statically (otherwise it would be printed out in the list above).

No, apparently there doesn't exist a similar option. But as I just figured out, you can do something else instead:

MPICH_CC=gcc OMPI_CC=gcc mpicc src/frontend.o src/ic.o src/masterworker.o src/models.o \
src/mpi_backend.o src/pltb.o src/pltb_frontend.o src/sequential.o \
/PATH/TO/libpll-sse3.a -lm -o pltb.out

So you just need to give the static library file path as a parameter to the compiler, just like some object file. Then it should link to that static library instead of the dynamic one.

[mhoff]

If someone wants to compile our program, he would need MPI and GCC installed. The PLL is linked statically (otherwise it would be printed out in the list above).

Do you mean "execute"?
Therefore we can provide partially linked binaries for MAC and Linux based systems?

thanks, in any case I guess a statically linked sequential and PThreads based versionshould be sufficient for most users, I didn't think about MPI issues

This way completely statically linked binaries could be produced.
@briehm do you want to test this?
In frontend.c there is a switch #define MPI_MASTER_WORKER 1 which should - hopefully - completely disable MPI includes, if turned off.

[briehm]

@mhoff:
Yes, if we provide the binaries, then they would need MPI and GCC in order to execute our program. Sorry for the misunderstanding!

In frontend.c there is a switch #define MPI_MASTER_WORKER 1 which should - hopefully - completely disable MPI includes, if turned off.

I've tested it, but it did not work. The compiled binary still had the mpi dynamic libs as dependencies.

[mhoff]

I've tested it, but it did not work. The compiled binary still had the mpi dynamic libs as dependencies.

Did you also compile without mpicc?

[stamatak]

I think we should not worry too much about pre-compiled MPI version
binaries, whoever can use an MPI code is probably also able to compile it,

alexis

On 30.06.2015 07:48, Michael Hoff wrote:

I've tested it, but it did not work. The compiled binary still had
the mpi dynamic libs as dependencies.

Did you also compile /without/ |mpicc|?

—
Reply to this email directly or view it on GitHub
#4 (comment).

Alexandros (Alexis) Stamatakis

Research Group Leader, Heidelberg Institute for Theoretical Studies
Full Professor, Dept. of Informatics, Karlsruhe Institute of Technology
Adjunct Professor, Dept. of Ecology and Evolutionary Biology, University
of Arizona at Tucson

www.exelixis-lab.org

[mhoff]

Thats probably true.
However I just applied some minor changes which enables us to build completely mpi-independent binaries. With Makefile.nompi and #define MPI_MASTER_WORKER 0 everything should work out of the box. @briehm can you verify this?
The only thing left to do is to combine Makefile.nompi and Makefile.static_call to produce 100% static (sequential) binaries.