Python does not find custom installed modules when running from a cronjob

I’m trying to set a python script to run every two minutes using a cronjob, the crontab line is below. The python script is called by a shell script, script.sh, posted below. This python script starts a selenium webdriver, gather some data and saves it to a csv file. A snippet is also bellow.

crontab

    */2 * * * * export DISPLAY=:0 && /home/me/workspace/project/script.sh > /home/me/workspace/project/cron.log 2>&1

The export DISPLAY=:0 is needed to run selenium in a cronjob, as seen here. This bit (> /home/me/workspace/project/cron.log 2>&1) logs output and errors to the cron.log file.

script.sh

    #!/bin/bash
    python3 /home/me/workspace/project/foo.py

foo.py

    from datetime import datetime
    from selenium import webdriver
    from pyvirtualdisplay import Display

    display = Display(visible=0, size=(800, 600))
    display.start()
    with webdriver.Chrome() as driver:
        #do some scraping and save to csv
        pass

When I run script.sh manually, everything runs smoothly, no errors. But when I set the crontab, the script fails. The cron.log file shows:

    Traceback (most recent call last):
    File "/home/me/workspace/project/foo.py", line 7, in <module>
        from selenium import webdriver
    ImportError: No module named 'selenium'

This says that selenium is not installed, but a pip3 freeze | grep selenium yields:

    $ pip3 freeze | grep selenium
    selenium==3.13.0

This shows that selenium is installed (as expected, because the script runs fine when started manually). The same thing happens with the other lib, pyvirtualdisplay, if I comment out the selenium line. So the problem is that, for some reason, python is not finding custom installed modules when running via cronjob. I’ve already tried replacing ‘python3’ by ‘/usr/bin/env python3’ or ‘/usr/bin/python3’ and other variations of it, to no avail.

If anyone could post a hint of a solution I’d appreciate, thanks.

Answer

#!/bin/bash
python3 /home/me/workspace/project/foo.py

I think this is your problem. @Martijn Pieters is spot on when he says it is a path issue. You need to ensure that the ‘python3’ you’re calling is the python3 that has selenium installed. Wherever you’re doing the pip3 freeze from, you need to ensure that is what the cron sees.

When I use cronjobs I have my anaconda virtualenv explicitly run the job, as follows:

/home/username/anaconda/envs/python3/bin/python /home/me/workspace/project/foo.py