When using the GCC driver, what makes a static lib “incompatible”?

Question

So what I am trying to do is on Ubuntu 14.04 (x86_64) I want to set up musl-libc based on the latest released 1.1.11 version which is available at this moment. What I did was to: Install multilib support for GCC: sudo apt-get --no-install-recommends install gcc-multilib Configure the libraries for 32-bit and 64-bit respectively and install them into separate folders:

Accepted Answer

The solutionIt turns out the solution is rather simple.We need to tell both the linker and the driver to use -m32 &#8230; I was that far before. However, it turns out that the missing piece was to pass the linker option to the driver via CFLAGS like this -Wl,-melf_i386.I am finally able to build and link the 32-bit executable on a multilib-enabled 64-bit host.NB: Below information is left in place for those who want to learn how I investigated the issue.Alright, so I investigated the issue a little further and the output gets more enlightening once you extract the object files. To reproduce what I am doing you may have to use Bash or a similar shell that allows for $(...) or you need to adjust the command lines accordingly.First off it&#8217;s important to have gcc-multilib and friends installed in order to target -m32 (i386-linux-gnu which happens to be an alias for i686-linux-gnu here).The code and the make fileI had the following make file:CC?=$(HOME)/bin/musl/bin64/musl-gccBLDARCH?=-m64CFLAGS+=-v $(BLDARCH)LDFLAGS+=-v -static $(BLDARCH)all: helloworldhelloworld: helloworld.cclean:    rm -f helloworldrebuild: clean all.PHONY: clean rebuild.NOTPARALLEL: rebuildand the following small helloworld.c:#include <stdio.h>int main(int argc, char** argv){    printf("Hello world!n");    return 0;}and my 32-bit musl-libc was installed to $HOME/bin/musl/{bin,include,lib}32 respectively. The 64-bit one was installed to $HOME/bin/musl/{bin,include,lib}64 respectively.Attempting to build with:make CC=$HOME/bin/musl/bin32/musl-gcc BLDARCH=-m32 rebuildalways failed with the same meaningless lines:/usr/bin/ld: skipping incompatible ~/bin/musl/lib32/libc.a when searching for -lc/usr/bin/ld: cannot find -lccollect2: error: ld returned 1 exit statusmake: *** [helloworld] Error 1Digging into the detailsSo after some contemplation I decided to re-do the steps done by the gcc driver manually.This meant to run roughly (I replaced all occurrences of my home folder with a ~):Compile: /usr/lib/gcc/x86_64-linux-gnu/4.8/cc1 -quiet -imultilib 32 -imultiarch i386-linux-gnu helloworld.c -nostdinc -isystem ~/bin/musl/include32 -isystem /usr/lib/gcc/x86_64-linux-gnu/4.8/include -quiet -dumpbase helloworld.c -m32 -mtune=generic -march=i686 -auxbase helloworld -version -fstack-protector -Wformat -Wformat-security -o /tmp/ccGmMuR1.sAssemble: as -v -v --32 -o /tmp/ccgRGlqf.o /tmp/ccGmMuR1.sLink: env COMPILER_PATH=/usr/lib/gcc/x86_64-linux-gnu/4.8/:/usr/lib/gcc/x86_64-linux-gnu/4.8/:/usr/lib/gcc/x86_64-linux-gnu/:/usr/lib/gcc/x86_64-linux-gnu/4.8/:/usr/lib/gcc/x86_64-linux-gnu/ LIBRARY_PATH=/usr/lib/gcc/x86_64-linux-gnu/4.8/32/:/usr/lib/gcc/x86_64-linux-gnu/4.8/../../../../lib32/:/lib/../lib32/:/usr/lib/../lib32/:/usr/lib/gcc/x86_64-linux-gnu/4.8/:/usr/lib/gcc/x86_64-linux-gnu/4.8/../../../:/lib/:/usr/lib/ COLLECT_GCC_OPTIONS="-v -v -static -m32 -o helloworld -specs=~/bin/musl/lib32/musl-gcc.specs -mtune=generic -march=i686" /usr/lib/gcc/x86_64-linux-gnu/4.8/collect2 -dynamic-linker /lib/ld-musl-i386.so.1 -nostdlib -static -z relro -o helloworld ~/bin/musl/lib32/crt1.o ~/bin/musl/lib32/crti.o /usr/lib/gcc/x86_64-linux-gnu/4.8/32/crtbegin.o -L~/bin/musl/lib32 -L /usr/lib/gcc/x86_64-linux-gnu/4.8/32/. /tmp/ccpL09mJ.o --start-group /usr/lib/gcc/x86_64-linux-gnu/4.8/32/libgcc.a /usr/lib/gcc/x86_64-linux-gnu/4.8/32/libgcc_eh.a -lc --end-group /usr/lib/gcc/x86_64-linux-gnu/4.8/32/crtend.o ~/bin/musl/lib32/crtn.oObviously this didn&#8217;t change the error message a bit just yet:/usr/bin/ld: skipping incompatible ~/bin/musl/lib32/libc.a when searching for -lc/usr/bin/ld: cannot find -lccollect2: error: ld returned 1 exit statusSo I took out the -lc from the command line for collect2 and created a folder ~/bin/musl/lib32/archive into which I extracted the whole libc.a generated by my musl-libc build attempt. I then instructed collect2 where to find the object files like this:Link: env COMPILER_PATH=/usr/lib/gcc/x86_64-linux-gnu/4.8/:/usr/lib/gcc/x86_64-linux-gnu/4.8/:/usr/lib/gcc/x86_64-linux-gnu/:/usr/lib/gcc/x86_64-linux-gnu/4.8/:/usr/lib/gcc/x86_64-linux-gnu/ LIBRARY_PATH=/usr/lib/gcc/x86_64-linux-gnu/4.8/32/:/usr/lib/gcc/x86_64-linux-gnu/4.8/../../../../lib32/:/lib/../lib32/:/usr/lib/../lib32/:/usr/lib/gcc/x86_64-linux-gnu/4.8/:/usr/lib/gcc/x86_64-linux-gnu/4.8/../../../:/lib/:/usr/lib/ COLLECT_GCC_OPTIONS="-v -v -static -m32 -o helloworld -specs=~/bin/musl/lib32/musl-gcc.specs -mtune=generic -march=i686" /usr/lib/gcc/x86_64-linux-gnu/4.8/collect2 -dynamic-linker /lib/ld-musl-i386.so.1 -nostdlib -static -z relro -o helloworld $HOME/bin/musl/lib32/crt1.o ~/bin/musl/lib32/crti.o /usr/lib/gcc/x86_64-linux-gnu/4.8/32/crtbegin.o -L~/bin/musl/lib32 -L /usr/lib/gcc/x86_64-linux-gnu/4.8/32/. /tmp/ccpL09mJ.o --start-group /usr/lib/gcc/x86_64-linux-gnu/4.8/32/libgcc.a /usr/lib/gcc/x86_64-linux-gnu/4.8/32/libgcc_eh.a --end-group /usr/lib/gcc/x86_64-linux-gnu/4.8/32/crtend.o ~/bin/musl/lib32/crtn.o $(find ~/bin/musl/lib32/archive -type f -name '*.o')This gist is taking out -lc and appending $(find ~/bin/musl/lib32/archive -type f -name '*.o').Which gave me a whole bunch of new, but more meaningful errors similar to the following ones:/usr/bin/ld: Warning: size of symbol `__init_ssp' changed from 1 in ~/bin/musl/lib32/archive/__libc_start_main.o to 65 in ~/bin/musl/lib32/archive/__stack_chk_fail.o/usr/bin/ld: Warning: size of symbol `__funcs_on_exit' changed from 126 in ~/bin/musl/lib32/archive/atexit.o to 1 in ~/bin/musl/lib32/archive/exit.o# more of those/usr/bin/ld: i386 architecture of input file `~/bin/musl/lib32/crt1.o' is incompatible with i386:x86-64 output/usr/bin/ld: i386 architecture of input file `~/bin/musl/lib32/crti.o' is incompatible with i386:x86-64 output# more of thoseThe plot thickens. Apparently the collect2 command gets the wrong idea about what to build. Which is odd considering the output for the governing environment variables:COMPILER_PATH=/usr/lib/gcc/x86_64-linux-gnu/4.8/:/usr/lib/gcc/x86_64-linux-gnu/4.8/:/usr/lib/gcc/x86_64-linux-gnu/:/usr/lib/gcc/x86_64-linux-gnu/4.8/:/usr/lib/gcc/x86_64-linux-gnu/LIBRARY_PATH=/usr/lib/gcc/x86_64-linux-gnu/4.8/32/:/usr/lib/gcc/x86_64-linux-gnu/4.8/../../../../lib32/:/lib/../lib32/:/usr/lib/../lib32/:/usr/lib/gcc/x86_64-linux-gnu/4.8/:/usr/lib/gcc/x86_64-linux-gnu/4.8/../../../:/lib/:/usr/lib/COLLECT_GCC_OPTIONS='-v' '-v' '-static' '-m32' '-o' 'helloworld' '-specs=~/bin/musl/lib32/musl-gcc.specs' '-mtune=generic' '-march=i686'&#8230; which I passed using env in my attempt to reproduce the conditions encountered by the collect2 wrapper when linking the libc.a.In order to find out more about the internals of the GNU compilers one needs to read https://gcc.gnu.org/onlinedocs/gccint/Unfortunately the part about collect2 doesn&#8217;t help us here. But the lengthy output of /usr/lib/gcc/x86_64-linux-gnu/4.8/collect2 --help looks promising.Just how could we smuggle our command line option to collect2?For now I was going to settle for whatever I could run manually. So I attempted to tell the linker which output format I expected. Based on the list of supported targets I was interested in elf_i386.Passing -melf_386 at the end of the previous line gave an interesting error output. Numerous referenced functions such as __vsyscall, __moddi3 and __divdi3 were undefined. That indicated they simply didn&#8217;t exist in the object files from the static lib or in any of the startup .o files for that matter:~/bin/musl/lib32/archive/aio.o: In function `cleanup':aio.c:(.text+0x5ad): undefined reference to `__vsyscall'aio.c:(.text+0x5bb): undefined reference to `__vsyscall'aio.c:(.text+0x5e8): undefined reference to `__vsyscall'aio.c:(.text+0x5f6): undefined reference to `__vsyscall'aio.c:(.text+0x61d): undefined reference to `__vsyscall'~/bin/musl/lib32/archive/aio.o:aio.c:(.text+0x62b): more undefined references to `__vsyscall' follow~/bin/musl/lib32/archive/cpow.o: In function `cpow':cpow.c:(.text+0x4f): undefined reference to `__mulxc3'~/bin/musl/lib32/archive/cpowf.o: In function `cpowf':cpowf.c:(.text+0x47): undefined reference to `__mulxc3'~/bin/musl/lib32/archive/cpowl.o: In function `cpowl':cpowl.c:(.text+0x4c): undefined reference to `__mulxc3'~/bin/musl/lib32/archive/sysconf.o: In function `sysconf':sysconf.c:(.text+0xcc): undefined reference to `__vsyscall'~/bin/musl/lib32/archive/__getdents.o: In function `__getdents':__getdents.c:(.text+0x13): undefined reference to `__vsyscall'~/bin/musl/lib32/archive/opendir.o: In function `opendir':opendir.c:(.text+0x37): undefined reference to `__vsyscall'~/bin/musl/lib32/archive/readdir.o: In function `readdir':readdir.c:(.text+0x1f): undefined reference to `__vsyscall'~/bin/musl/lib32/archive/__init_tls.o: In function `__init_tls':__init_tls.c:(.text+0x136): undefined reference to `__vsyscall6'__init_tls.c:(.text+0x16e): undefined reference to `__vsyscall'As I had already clarified in my question, the object files from the archive all stated that they were elf32-i386.The functions __vsyscall and __vsyscall6 should end up in a file called syscall.o given the source file for i386 in musl-libc: src/internal/i386/syscall.s. Let&#8217;s verify that first. Since there is also a file src/misc/syscall.c the name might be different though. Four files have syscall in the file name:__syscall_cp.osyscall_cp.osyscall.osyscall_ret.oQuerying those files using nm gave:$ nm -s $(ls |grep syscall)__syscall_cp.o:00000000 t sccp         U __syscall00000005 T __syscall_cp00000000 W __syscall_cp_csyscall_cp.o:         U __cancel00000008 T __cp_begin00000035 T __cp_cancel00000030 T __cp_end00000000 T __syscall_cp_asmsyscall.o:00000000 T syscall         U __syscall_ret         U __vsyscall6syscall_ret.o:         U __errno_location00000000 T __syscall_retSymbols with the symbol type U are undefined and therefore expected by the linker to come from outside (external to each object file).A final $ nm --defined-only *.o ../*.o|grep vsyscall was what was needed to verify that those symbols were indeed missing from the libc.a.So the cross-built libc.a what was faulty after all. Back to the drawing board.I hope this description helps others to figure out similar issues and look behind the scenes in GCC.The saga continuesI was really surprised to see:$ nm --defined-only ../libc.a |grep -B 2 vsyscallsyscall.o:0000004b T __syscall00000000 T __vsyscall00000031 T __vsyscall6but for the extracted object files the corresponding command (nm --defined-only *.o ../*.o|grep -B 2 vsyscall) would yield no output.So inside the libc.a somehow nm sees the two symbols, but after extracting them they disappear? Odd.Let&#8217;s look for syscall.o in the libc.a:$ nm ../libc.a |grep ^syscallsyscall.o:syscall_ret.o:syscall.o:syscall_cp.o:Whoa? So syscall.o exists twice inside the static library? Well that looks like it may just be the error cause we&#8217;re looking for. And it certainly does explain why the symbols disappear. Likely the latter syscall.o overwrites the one first extracted when running ar x ....Confirming:$ nm ../libc.a |grep -A 4 ^syscall.osyscall.o:0000004b T __syscall         U __sysinfo00000000 T __vsyscall00000031 T __vsyscall6--syscall.o:00000000 T syscall         U __syscall_ret         U __vsyscall6and looking into the musl-libc source tree after doing the 32-bit build:$ find . -type f -name 'syscall.o' -exec nm {} +./src/internal/syscall.o:0000004b T __syscall         U __sysinfo00000000 T __vsyscall00000031 T __vsyscall6./src/misc/syscall.o:00000000 T syscall         U __syscall_ret         U __vsyscall6Copying the former into the lib32/archive directory under a name that doesn&#8217;t collide with existing names gives more errors on other functions, suggesting other object files may also exist as duplicates inside the generated libc.a.Which ones are duplicates?$ diff <(nm libc.a|grep ':$'|cut -f 1 -d :|sort) <(nm libc.a|grep ':$'|cut -f 1 -d :|sort -u)--- /dev/fd/63  2015-10-05 23:58:53.683804823 +0000+++ /dev/fd/62  2015-10-05 23:58:53.683804823 +0000@@ -131,7 +131,6 @@ clogl.o clog.o clone.o-clone.o closedir.o close.o cnd_broadcast.o@@ -1115,7 +1114,6 @@ __syscall_cp.o syscall_cp.o syscall.o-syscall.o syscall_ret.o sysconf.o sysinfo.oThis way we see clone.o and syscall.o are affected as duplicates. Which indicates that some object files are missing altogether from the libc.a given undefined references to the following symbols:__divdi3__moddi3__mulxc3__tls_get_new__udivdi3__umoddi3These names happen to coincide with the ones from the Integer library routines listed for GCC. Which makes me think I am missing one library provided by GCC which to link. Like libgcc?! &#8230;I have package lib32gcc-4.8-dev which means I should have the required file:$ apt-file list lib32gcc-4.8-dev|grep -E 'libgcc.*.a'lib32gcc-4.8-dev: /usr/lib/gcc/x86_64-linux-gnu/4.8/32/libgcc.alib32gcc-4.8-dev: /usr/lib/gcc/x86_64-linux-gnu/4.8/32/libgcc_eh.aAfter the previous step I decided to set up a x86_32 version on Ubuntu 14.04 that was essentially at the same patch level.I then compared the libgcc.a from the x86_32 machine with that from the x86_64 machine. They turned out to be almost identical except for the &#8220;symbol value&#8221; (in nm lingo) of a handful of functions.Also, since the symbols were in the static library I attempted to link against the static library again. This worked only with -melf_i386 on the linker (collect2) command line.After trying to use LDFLAGS and noticing that those were also passed to cc1, I set appended -Wl,-melf_i386 to the CFLAGS and now it worked. Brilliant.As a side-note: I also swapped the /usr/lib/gcc/x86_64-linux-gnu/4.8/collect2 in the command line for ld of which collect2 is supposed to be a wrapper. The error output was identical.

Advertisement

Answer

The solution

The code and the make file

Digging into the details

The saga continues

Which ones are duplicates?