Cannot simultaneously run python 2 and 3 due to import errors

I am trying to be able to run python2 and python3 simultaneously on a server that I do not have root access to and which does not have a recent python3 implementation. This works fine on my own server which I do have root access to, but I can’t figure out what I am doing wrong.

For reference, the python2 implementation has been installed by administrators on the centos/rocks system in question, and they have both version 2.6.6 and 2.7.5. They also installed python 3.3.2, but there are some things in version 3.4 that I use, so I custom installed version 3.4.3 and put it in my PATH, LIBPATH, and PYTHONPATH. I use that python version exclusively for my own code, and I install libraries there with pip. I have created my own custom rocks module for the python3 version that sets the PATH, LIBPATH and LD_LIBRARY_PATH.

Everything works fine when I am just running python3 code. The problem comes when I try and simultaneously run python3 and python2 code in the same session.

I have written code in python3 that makes a system call to run a python2 program¹. The problem is that in order to run my python3 code, I have my PYTHONPATH set to only contain python3 libraries. If python2 libraries are in that PATH, I get errors similar to this:

Failed to import the site module
Traceback (most recent call last):
  File "/opt/scipy/2.7/lib/python2.7/site-packages/site.py", line 73, in <module>
    __boot()
  File "/opt/scipy/2.7/lib/python2.7/site-packages/site.py", line 2, in __boot
    import sys, imp, os, os.path   
  File "/oasis/projects/nsf/sua137/peanut/usr/lib/python3.4/imp.py", line 22, in <module>
    from importlib import util
  File "/oasis/projects/nsf/sua137/peanut/usr/lib/python3.4/importlib/util.py", line 12, in <module>
    from contextlib import contextmanager
  File "/oasis/projects/nsf/sua137/peanut/usr/lib/python3.4/contextlib.py", line 4, in <module>
    from collections import deque
  File "/oasis/projects/nsf/sua137/peanut/usr/lib/python3.4/collections/__init__.py", line 17, in <module>
    from reprlib import recursive_repr as _recursive_repr
  File "/opt/biotools/qiime/lib/python2.7/site-packages/reprlib/__init__.py", line 8, in <module>
    raise ImportError('Cannot import module from python-future source folder')
ImportError: Cannot import module from python-future source folder

However, if I strip out all of the python2 stuff from the PYTHONPATH, then python2 scripts fail with errors like this:

File "/oasis/projects/nsf/sua137/peanut/usr/lib/python3.4/site.py", line 176
file=sys.stderr)
    ^
SyntaxError: invalid syntax

The only way I have found to avoid this is to include an explicit PYTHONPATH in the call to the python2 code. That does work, but it is hugely cumbersome and ugly.

I used to keep all of my python3 stuff in the PYTHON3PATH, and the python2 stuff in the PYTHONPATH, but now python3 appears to completely ignore the PYTHON3PATH and only use the PYTHONPATH.

Again, this problem doesn’t occur on my own personal server, which runs arch linux and has python 3.4 and python 2.7 running side-by-side happily even though my PYTHONPATH contains directories for both.

I am sure I am doing something stupid here, but I don’t know what it is.

¹ Footnote to stave off silly comments: They python2 script is not written by me, the python3 code is a simple job management system that coordinates multi-threading none-parallel code on a remote machine. It is supposed to run any standalone program in parallel, but it is failing with python2 scripts. So I can’t just switch one or the other to either version 2 or 3, and I can’t merge the two by using 2to3 and importing.

Answer

Calling python with the explicit path or version resolved the issue. For example:

$ python3.x
$ path_to/python

Advertisement

Answer