Upgrading `pip` removes other python’s pips

On a CentOS 7 system, I have multiple versions of Python installed, each with their own version of pip:

# head -n1 /usr/local/bin/pip3.*
==> /usr/local/bin/pip3.6 <==
#!/usr/bin/python3

==> /usr/local/bin/pip3.7 <==
#!/usr/local/bin/python3.7

==> /usr/local/bin/pip3.8 <==
#!/usr/local/bin/python3.8

When I ask pip3.8 to upgrade itself, it removes the installed pip3.7:

# pip3.8 install --upgrade pip
Collecting pip
  Using cached https://files.pythonhosted.org/packages/54/0c/d01aa759fdc501a58f431eb594a17495f15b88da142ce14b5845662c13f3/pip-20.0.2-py2.py3-none-any.whl
Installing collected packages: pip
  Found existing installation: pip 19.2.3
    Uninstalling pip-19.2.3:
      Successfully uninstalled pip-19.2.3
Successfully installed pip-20.0.2


# head -n1 /usr/local/bin/pip3.*
==> /usr/local/bin/pip3.6 <==
#!/usr/bin/python3

==> /usr/local/bin/pip3.8 <==
#!/usr/local/bin/python3.8

Why is it doing this, and how can I prevent it?

UPDATES:

• The lib paths are different for the two installations, as shown here:

# python3.7 -c 'import sys; print(sys.path)'
['', '/usr/local/lib/python37.zip', '/usr/local/lib/python3.7', '/usr/local/lib/python3.7/lib-dynload', '/usr/local/lib/python3.7/site-packages']
# python3.8 -c 'import sys; print(sys.path)'
['', '/usr/local/lib/python38.zip', '/usr/local/lib/python3.8', '/usr/local/lib/python3.8/lib-dynload', '/usr/local/lib/python3.8/site-packages']

• It is not bidirectional – upgrading pip3.7 does not remove pip3.8.

• I believe the library gets upgraded correctly and leaves the version 3.7 library in place, it’s just the shell wrapper script that’s deleted. Here’s after the pip3.8 upgrade:

# python3.7 -m pip --version
pip 20.0.2 from /usr/local/lib/python3.7/site-packages/pip (python 3.7)
# python3.8 -m pip --version
pip 20.0.2 from /usr/local/lib/python3.8/site-packages/pip (python 3.8)
# pip3.7 --version
bash: pip3.7: command not found
# pip3.8 --version
pip 20.0.2 from /usr/local/lib/python3.8/site-packages/pip (python 3.8)

• Doing pip3.7 install --upgrade pip does not remove /usr/local/bin/pip3.6, so it’s not the case that it always removes previous versions.

• For full reproducibility, and to show that I’m starting with a fairly pristine system, here’s a Gist containing my Dockerfile text: https://gist.github.com/kenahoo/a1104f9cb84694fbd5ec9d6d560a885e . It fails on the RUN pip3.7 install setuptools numpy pandas line because pip3.7 has gone missing.

• It doesn’t matter whether I upgrade using python3.8 -m pip install --upgrade pip or pip3.8 install --upgrade pip, both of them end up removing the /usr/local/bin/pip3.7 wrapper script.

Answer

I believe I found the issue.

In short, the pipX.Y console script is set to the version of the Python interpreter used to build the pip‘s wheel, instead of the version of the Python interpreter used to install it.

• https://github.com/pypa/pip/issues/8010

For example take any pip installed in any Python that is not 3.8 (in my case it’s Python 3.6) and use it to download pip itself:

$ /path/to/pythonX.Y -m pip download pip

This should give you a wheel file for example pip-20.0.2-py2.py3-none-any.whl, now unzip it:

$ /path/to/pythonX.Y -m zipfile -e pip-20.0.2-py2.py3-none-any.whl .

Now look at the content of pip-20.0.2.dist-info/entry_points.txt:

$ cat pip-20.0.2.dist-info/entry_points.txt 
[console_scripts]
pip = pip._internal.cli.main:main
pip3 = pip._internal.cli.main:main
pip3.8 = pip._internal.cli.main:main

So there is an entry for a console script pip3.8 even though I have Python 3.6. This is obviously wrong. And for example if I indeed had an actual pip3.8 script then this file would be deleted when uninstalling the pip associated with the Python 3.6, for example to upgrade it.

The root of the issue can be seen here for example:

• https://github.com/pypa/pip/blob/19.2.3/setup.py#L76

    entry_points={
        "console_scripts": [
            "pip=pip._internal:main",
            "pip%s=pip._internal:main" % sys.version_info[:1],
            "pip%s.%s=pip._internal:main" % sys.version_info[:2],
        ],
    },

This line pip%s.%s=pip._internal:main" % sys.version_info[:2] gets actually written down definitely when building the wheel, and I assume the wheel we downloaded earlier was built with Python 3.8.

---

That bug is (at least partially) known to pip‘s maintainers, and not sure it will get fixed (probably not worth it).

Anyway, one should always use the explicit /path/to/pythonX.Y -m pip instead. The pip* scripts are just shortcuts that are here for convenience. They are somewhat useful from an interactive command line to save some keystrokes and be able to work faster. But in a file, anything from documentation, to shell scripts, or Dockerfiles, I am the opinion that one should always use the explicit expanded versions. For example I always write rm --recursive instead of rm -r, etc.

Additionally in the one particular case of Python‘s pip, it makes sense no matter what:

• https://snarky.ca/why-you-should-use-python-m-pip/