Skip to content
Advertisement

compile official doc “Embedding Python in Another Application” example failed

I am trying to compile and run the example from https://docs.python.org/3/extending/embedding.html#very-high-level-embedding , but failed.

My environment is Ubuntu 20.04.2 LTS, with system shipped python3.8(statically built), libpython3-dev and libpython3.8-dev packages installed.

What I’ve tried:

main.c :

#define PY_SSIZE_T_CLEAN
#include <Python.h>

int
main(int argc, char *argv[])
{
    wchar_t *program = Py_DecodeLocale(argv[0], NULL);
    if (program == NULL) {
        fprintf(stderr, "Fatal error: cannot decode argv[0]n");
        exit(1);
    }
    Py_SetProgramName(program);  /* optional but recommended */
    Py_Initialize();
    PyRun_SimpleString("from time import time,ctimen"
                       "print('Today is', ctime(time()))n");
    if (Py_FinalizeEx() < 0) {
        exit(120);
    }
    PyMem_RawFree(program);
    return 0;
}

From https://docs.python.org/3/extending/embedding.html#compiling-and-linking-under-unix-like-systems, get gcc flags.

tian@tian-B250M-Wind:~$ python3-config --cflags
-I/usr/include/python3.8 -I/usr/include/python3.8  -Wno-unused-result -Wsign-compare -g -fdebug-prefix-map=/build/python3.8-4wuY7n/python3.8-3.8.10=. -specs=/usr/share/dpkg/no-pie-compile.specs -fstack-protector -Wformat -Werror=format-security  -DNDEBUG -g -fwrapv -O3 -Wall
tian@tian-B250M-Wind:~$ python3-config --ldflags
-L/usr/lib/python3.8/config-3.8-x86_64-linux-gnu -L/usr/lib  -lcrypt -lpthread -ldl  -lutil -lm -lm

(I don’t know why python3-config output has some duplicated values, that’s not a typing mistake)

gcc {copy cflags output} -o main main.c /path_to_libpython/libpython3.8.a {copy ld flags output}:

gcc -I/usr/include/python3.8 -I/usr/include/python3.8  -Wno-unused-result -Wsign-compare -g -fdebug-prefix-map=/build/python3.8-4wuY7n/python3.8-3.8.10=. -specs=/usr/share/dpkg/no-pie-compile.specs -fstack-protector -Wformat -Werror=format-security  -DNDEBUG -g -fwrapv -O3 -Wall -o main main.c /usr/lib/python3.8/config-3.8-x86_64-linux-gnu/libpython3.8.a -L/usr/lib/python3.8/config-3.8-x86_64-linux-gnu -L/usr/lib  -lcrypt -lpthread -ldl  -lutil -lm -lm

make test1 gives error, nearly all related to -fPIE error:

/usr/bin/ld: /tmp/ccdMZ2Yk.o: relocation R_X86_64_32 against `.rodata.str1.8' can not be used when making a PIE object; recompile with -fPIE
/usr/bin/ld: /usr/lib/python3.8/config-3.8-x86_64-linux-gnu/libpython3.8.a(obmalloc.o): relocation R_X86_64_32 against `.text.hot' can not be used when making a PIE object; recompile with -fPIE
/usr/bin/ld: /usr/lib/python3.8/config-3.8-x86_64-linux-gnu/libpython3.8.a(initconfig.o): relocation R_X86_64_32 against symbol `_PyRuntime' can not be used when making a PIE object; recompile with -fPIE
/usr/bin/ld: /usr/lib/python3.8/config-3.8-x86_64-linux-gnu/libpython3.8.a(pathconfig.o): relocation R_X86_64_32 against symbol `_Py_path_config' can not be used when making a PIE object; recompile with -fPIE
/usr/bin/ld: /usr/lib/python3.8/config-3.8-x86_64-linux-gnu/libpython3.8.a(preconfig.o): relocation R_X86_64_32 against `.rodata.str1.1' can not be used when making a PIE object; recompile with -fPIE
/usr/bin/ld: /usr/lib/python3.8/config-3.8-x86_64-linux-gnu/libpython3.8.a(pylifecycle.o): relocation R_X86_64_32 against symbol `_PyRuntime' can not be used when making a PIE object; recompile with -fPIE
/usr/bin/ld: /usr/lib/python3.8/config-3.8-x86_64-linux-gnu/libpython3.8.a(pystate.o): relocation R_X86_64_32 against `.rodata.str1.8' can not be used when making a PIE object; recompile with -fPIE
/usr/bin/ld: /usr/lib/python3.8/config-3.8-x86_64-linux-gnu/libpython3.8.a(pythonrun.o): relocation R_X86_64_32 against symbol `_PyParser_Grammar' can not be used when making a PIE object; recompile with -fPIE
/usr/bin/ld: /usr/lib/python3.8/config-3.8-x86_64-linux-gnu/libpython3.8.a(pytime.o): relocation R_X86_64_32S against symbol `PyFloat_Type' can not be used when making a PIE object; recompile with -fPIE
.....

I saw there is a /usr/lib/python3.8/config-3.8-x86_64-linux-gnu/libpython3.8-pic.a and tried link it instead but also failed (log).

Later I tried a Docker image(python:3.10.5-bullseye) with share build python and succeeded.

root@tian-B250M-Wind:/# python3-config --cflags
-I/usr/local/include/python3.10 -I/usr/local/include/python3.10  -Wno-unused-result -Wsign-compare  -DNDEBUG -g -fwrapv -O3 -Wall
root@tian-B250M-Wind:/# python3-config --ldflags
 -L/usr/local/lib  -lcrypt -lpthread -ldl  -lutil -lm -lm

gcc -I/usr/local/include/python3.10 -I/usr/local/include/python3.10  -Wno-unused-result -Wsign-compare  -DNDEBUG -g -fwrapv -O3 -Wall -o main main.c -L/usr/local/lib  -lcrypt -lpthread -ldl  -lutil -lm -lm -lpython3.10

(I add -lpython3.10 in the end)

It compiles and ./main :

root@tian-B250M-Wind:/# ./main
Today is Thu Jul 14 10:39:20 2022

What’s wrong with my compilation for the Ubuntu system shipped, python3.8 one?

Anyway, I just want to validate that I can link the static python library libpythonx.y.a . So if anyone can make that work on a fresh installed machine or with non system shipped python (e.g. self-built static python), I’d like to try.

Advertisement

Answer

I made more experiments on both system shipped python and self built python libraries. For self bulilt pythons, it’s easy to compile successfully, either for shared built python or statically built python.

For system shipped python, shared link is easy and what I’ve missed for static link is that I need to link 2 extra libs -lexpat and -lz. Also there are 2 ways for system shipped python to link successfully:

  1. Add -no-pie, and -lexpat -lz.
  2. No -no-pie, use libpython3.8-pic.a, and -lexpat -lz.

Check the py38_system_static target below.

makefile:

py38_system_failed:
    gcc -I/usr/include/python3.8 -I/usr/include/python3.8  -Wno-unused-result -Wsign-compare -g -fdebug-prefix-map=/build/python3.8-4wuY7n/python3.8-3.8.10=. -specs=/usr/share/dpkg/no-pie-compile.specs -fstack-protector -Wformat -Werror=format-security  -DNDEBUG -g -fwrapv -O3 -Wall -o main main.c /usr/lib/python3.8/config-3.8-x86_64-linux-gnu/libpython3.8.a -L/usr/lib/python3.8/config-3.8-x86_64-linux-gnu -L/usr/lib  -lcrypt -lpthread -ldl  -lutil -lm -lm

py38_system_shared:
    # 36K main
    gcc -I/usr/include/python3.8 -I/usr/include/python3.8  -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -o main main.c -L/usr/lib/python3.8/config-3.8-x86_64-linux-gnu -L/usr/lib -lpython3.8 -lcrypt -lpthread -ldl  -lutil -lm -lm 

py38_system_static:
    # 5.5M main 
    gcc -I/usr/include/python3.8 -I/usr/include/python3.8 -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -no-pie -O3 -Wall -o main main.c /usr/lib/python3.8/config-3.8-x86_64-linux-gnu/libpython3.8.a -L/usr/lib/python3.8/config-3.8-x86_64-linux-gnu -L/usr/lib -lcrypt -lpthread -ldl  -lutil -lm -lm -lexpat -lz
    gcc -I/usr/include/python3.8 -I/usr/include/python3.8 -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -o main main.c /usr/lib/python3.8/config-3.8-x86_64-linux-gnu/libpython3.8-pic.a -L/usr/lib/python3.8/config-3.8-x86_64-linux-gnu -L/usr/lib -lcrypt -lpthread -ldl  -lutil -lm -lm -lexpat -lz

docker_py310_share:
    # 21K main
    gcc -I/usr/local/include/python3.10 -I/usr/local/include/python3.10  -Wno-unused-result -Wsign-compare  -DNDEBUG -g -fwrapv -O3 -Wall -o main main.c -L/usr/local/lib  -lcrypt -lpthread -ldl  -lutil -lm -lm -lpython3.10

py36_static:
    # 15M main
    gcc -I/home/tian/py3.6.12_static/include/python3.6m -I/home/tian/py3.6.12_static/include/python3.6m  -Wno-unused-result -Wsign-compare  -DNDEBUG -g -fwrapv -O3 -Wall -o main main.c -L/home/tian/py3.6.12_static/lib/python3.6/config-3.6m-x86_64-linux-gnu -L/home/tian/py3.6.12_static/lib -lpython3.6m -lpthread -ldl  -lutil -lm  -Xlinker -export-dynamic

py36_shared:
    # 36K main
    gcc -I/home/tian/py3.6.12_shared/include/python3.6m -I/home/tian/py3.6.12_shared/include/python3.6m  -Wno-unused-result -Wsign-compare  -DNDEBUG -g -fwrapv -O3 -Wall -o main main.c -L/home/tian/py3.6.12_shared/lib -lpython3.6m -lpthread -ldl  -lutil -lm  -Xlinker -export-dynamic

py39_static:
    # 22M main
    gcc -I/home/tian/py3.9.13_static/include/python3.9 -I/home/tian/py3.9.13_static/include/python3.9  -Wno-unused-result -Wsign-compare  -DNDEBUG -g -fwrapv -O3 -Wall -o main main.c  -L/home/tian/py3.9.13_static/lib/python3.9/config-3.9-x86_64-linux-gnu -L/home/tian/py3.9.13_static/lib  -lpython3.9 -lcrypt -lpthread -ldl  -lutil -lm -lm 
    gcc -I/home/tian/py3.9.13_static/include/python3.9 -I/home/tian/py3.9.13_static/include/python3.9  -Wno-unused-result -Wsign-compare  -DNDEBUG -g -fwrapv -O3 -Wall -o main2 main.c /home/tian/py3.9.13_static/lib/libpython3.9.a -lcrypt -lpthread -ldl  -lutil -lm -lm 
    
py39_shared:
    # 36K main
    gcc -I/home/tian/py3.9.13_shared/include/python3.9 -I/home/tian/py3.9.13_shared/include/python3.9  -Wno-unused-result -Wsign-compare  -DNDEBUG -g -fwrapv -O3 -Wall -o main main.c -L/home/tian/py3.9.13_shared/lib  -lpython3.9 -lcrypt -lpthread -ldl  -lutil -lm -lm 

clean:
    rm main && rm main2
User contributions licensed under: CC BY-SA
3 People found this is helpful
Advertisement