I am using CentOS. I didn’t set up the server and the environment is quite old.
I tried to debug a multithreaded server program and some error showed up.
Starting program: ./battleTrying host libthread_db library: libthread_db.so.1.Host libthread_db.so.1 resolved to: /lib64/libthread_db.so.1.td_ta_new failed: application not linked with libthreadthread_db_load_search returning 0warning: Could not load shared library symbols for linux-vdso.so.1.Do you need "set solib-search-path" or "set sysroot"?Trying host libthread_db library: libthread_db.so.1.Host libthread_db.so.1 resolved to: /lib64/libthread_db.so.1.td_ta_new failed: versions of libpthread and libthread_db do not matchTrying host libthread_db library: /lib64/libthread_db.so.1.td_ta_new failed: versions of libpthread and libthread_db do not matchthread_db_load_search returning 0warning: Unable to find libthread_db matching inferior's thread library, thread debugging will not be available.Then I realize that the problem is that the version of libpthread shared library I link doesn’t match the version of libthread_db shared library which gdb uses when debugging.
I read these 2 related questions Unable to Debug Multi-Threaded Application with gdb, GDB debugging warnings .
Then I tried to figure out what version of each is by using file command (Please tell me if using file to check the differences is correct. Anyway, I found some differences.)
ldd battle linux-vdso.so.1 (0x00007fffcafff000) libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f390c675000) ls -l libpthread* -rwxr-xr-x 1 root root 143280 Apr 9 2019 libpthread-2.12.so -rwxr-xr-x 1 root root 806517 Dec 2 2018 libpthread-2.17.so lrwxrwxrwx 1 root root 18 Jul 29 07:35 libpthread.so.0 -> libpthread-2.17.sols -l libthread_db* -rwxr-xr-x 1 root root 34488 Apr 9 2019 libthread_db-1.0.so lrwxrwxrwx 1 root root 19 Jul 29 07:35 libthread_db.so.1 -> libthread_db-1.0.sofile libpthread-2.17.so libpthread-2.17.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.16, not strippedfile libthread_db-1.0.so libthread_db-1.0.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.18, not strippedfile libpthread-2.12.so libpthread-2.12.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.18, not strippedSo my progam is linked to libpthread-2.17.so , which is for for GNU/Linux 2.6.16, and gdb uses libthread_db-1.0.so, which is for for GNU/Linux 2.6.18, mismatch? (Well I am not sure if I can compare them based on this information, can I?)
Does it mean that now I need to download a libthread_db-1.0.so which is for for GNU/Linux 2.6.16 and link libthread_db.so.1 to it? Where can I download it from?
By the way, I also feel strange that for libpthread-2.17.so and libpthread-2.12.so on my server , 2.17 > 2.12, the former should be a newer version. But it’s for a lower Linux version (2.6.16), maybe someone just copy it and rename the file? I don’t know. 🙁
Advertisement
Answer
Your question is the exact duplicate of the two answers you’ve already found: someone installed GLIBC-2.17 onto your system (which likely had GLIBC-2.12 originally), but failed to install matching libthread_db.so.1.
The output of file — GNU/Linux 2.6.18 is a red herring and is irrelevant.
The good news is that updating libpthread_db is risk-free — if you screw up, the worst that can happen is that GDB will not work for multi-threaded programs, but since that is the current state, you can’t make it any worse.
But you should be able to make it better:
- download GLIBC-2.17 source, configure and build it (but don’t
make install, or you’ll likely brick your system). - as part of the build, there should now exist
build/nptl_db/libthread_db.so.1 - test it by pointing GDB to it with
set libthread-db-search-path /path/to/build/nptl_dband trying to debug your program again.
If that works, you can overwrite the non-working /lib64/libthread_db.so.1 with the one you just built.
Update:
Is there any command I can use to grab some version information of these shared library files?
For libpthread*.so.0, this is quite easy:
gdb -q /lib/x86_64-linux-gnu/libpthread.so.0Reading symbols from /lib/x86_64-linux-gnu/libpthread.so.0(gdb) x/s nptl_version0x16037 <nptl_version>: "2.30"For libthread_db.so.1 it’s quite a bit harder: you need to disassemble td_ta_new, and look for memcmp call, which could be inlined:
gdb -q /lib/x86_64-linux-gnu/libthread_db.so.1Reading symbols from /lib/x86_64-linux-gnu/libthread_db.so.1(gdb) disas td_ta_newDump of assembler code for function td_ta_new: 0x0000000000002270 <+0>: push %r13 0x0000000000002272 <+2>: push %r12 0x0000000000002274 <+4>: push %rbp 0x0000000000002275 <+5>: mov %rsi,%rbp 0x00000000000022d6 <+102>: callq 0x2110 <ps_pdread@plt> 0x00000000000022db <+107>: mov %eax,%r12d 0x00000000000022de <+110>: test %eax,%eax 0x00000000000022e0 <+112>: jne 0x2318 <td_ta_new+168> 0x00000000000022e2 <+114>: cmpl $0x30332e32,0x13(%rsp) 0x00000000000022ea <+122>: je 0x2340 <td_ta_new+208> 0x00000000000022ec <+124>: mov $0x16,%r12d 0x00000000000022f2 <+130>: mov 0x18(%rsp),%rax 0x00000000000022f7 <+135>: xor %fs:0x28,%rax 0x0000000000002300 <+144>: jne 0x2388 <td_ta_new+280> 0x0000000000002306 <+150>: add $0x28,%rsp 0x000000000000230a <+154>: mov %r12d,%eax 0x000000000000230d <+157>: pop %rbx 0x000000000000230e <+158>: pop %rbp 0x000000000000230f <+159>: pop %r12 0x0000000000002311 <+161>: pop %r13 0x0000000000002313 <+163>: retqHere, at address 0x022e2 is an inlined call to memcmp("2.30", <data-read-from-nptl_version>, 5).