Why is subprocess ignoring PATH, and how can I change this?

I need to change which program is called by a Python application. Unfortunately I cannot change the Python code. I can only change the calling environment (in particular, PATH). But unfortunately Python’s subprocess module seems to ignore PATH (at least under certain circumstances).

How can I force Python to respect PATH when searching which binary to invoke?

To illustrate the problem, here’s an MVCE. The actual Python application is using subprocess.check_output(['nvidia-smi', '-L']), but the following simplified code shows the same behaviour.

Create test.py:

import os
from subprocess import run

run(['which', 'whoami'])
run(['/usr/bin/env', 'whoami'])
run(['whoami'])

os.execvp('whoami', ['whoami'])

Now create a local whoami script and execute test.py:

echo 'echo foobar' >whoami
chmod +x whoami
PATH=.:$PATH python3 test.py

On my system¹ this prints:

./whoami
foobar
konrad
konrad

I expect this code to always print foobar instead of konrad.

My MVCE includes the os.execvp call because the subprocess documentation states that

On POSIX, the class uses os.execvp()-like behavior to execute the child program.

Needless to say, the actual execvp POSIX API, called from C, does respect PATH, so this is a Python specific issue.

---

¹ Ubuntu 18.04.2 LTS, Python 3.6.9.

Answer

as per my comment, this is due to Python’s implementation of execvp not being consistent with POSIX execvp semantics. in particular Python doesn’t respond to ENOEXEC errors by interpreting the file as a shell script and needs an explicit shebang.

creating the file as:

printf '#!/bin/shnecho foobarn' > ./whoami

causes things to work as expected

note that this has been known about for a while: https://bugs.python.org/issue19948