I need to change which program is called by a Python application. Unfortunately I cannot change the Python code. I can only change the calling environment (in particular, PATH). But unfortunately Python’s subprocess module seems to ignore PATH (at least under certain circumstances).
How can I force Python to respect PATH when searching which binary to invoke?
To illustrate the problem, here’s an MVCE. The actual Python application is using subprocess.check_output(['nvidia-smi', '-L']), but the following simplified code shows the same behaviour.
Create test.py:
import os
from subprocess import run
run(['which', 'whoami'])
run(['/usr/bin/env', 'whoami'])
run(['whoami'])
os.execvp('whoami', ['whoami'])
Now create a local whoami script and execute test.py:
echo 'echo foobar' >whoami chmod +x whoami PATH=.:$PATH python3 test.py
On my system1 this prints:
./whoami foobar konrad konrad
I expect this code to always print foobar instead of konrad.
My MVCE includes the os.execvp call because the subprocess documentation states that
On POSIX, the class uses
os.execvp()-like behavior to execute the child program.
Needless to say, the actual execvp POSIX API, called from C, does respect PATH, so this is a Python specific issue.
1 Ubuntu 18.04.2 LTS, Python 3.6.9.
Advertisement
Answer
as per my comment, this is due to Python’s implementation of execvp not being consistent with POSIX execvp semantics. in particular Python doesn’t respond to ENOEXEC errors by interpreting the file as a shell script and needs an explicit shebang.
creating the file as:
printf '#!/bin/shnecho foobarn' > ./whoami
causes things to work as expected
note that this has been known about for a while: https://bugs.python.org/issue19948