I’m trying to set a python script to run every two minutes using a cronjob, the crontab line is below. The python script is called by a shell script, script.sh, posted below. This python script starts a selenium webdriver, gather some data and saves it to a csv file. A snippet is also bellow.
crontab
*/2 * * * * export DISPLAY=:0 && /home/me/workspace/project/script.sh > /home/me/workspace/project/cron.log 2>&1
The export DISPLAY=:0 is needed to run selenium in a cronjob, as seen here. This bit (> /home/me/workspace/project/cron.log 2>&1) logs output and errors to the cron.log file.
script.sh
#!/bin/bash python3 /home/me/workspace/project/foo.py
foo.py
from datetime import datetime from selenium import webdriver from pyvirtualdisplay import Display display = Display(visible=0, size=(800, 600)) display.start() with webdriver.Chrome() as driver: #do some scraping and save to csv pass
When I run script.sh manually, everything runs smoothly, no errors. But when I set the crontab, the script fails. The cron.log file shows:
Traceback (most recent call last): File "/home/me/workspace/project/foo.py", line 7, in <module> from selenium import webdriver ImportError: No module named 'selenium'
This says that selenium is not installed, but a pip3 freeze | grep selenium yields:
$ pip3 freeze | grep selenium selenium==3.13.0
This shows that selenium is installed (as expected, because the script runs fine when started manually). The same thing happens with the other lib, pyvirtualdisplay, if I comment out the selenium line. So the problem is that, for some reason, python is not finding custom installed modules when running via cronjob. I’ve already tried replacing ‘python3’ by ‘/usr/bin/env python3’ or ‘/usr/bin/python3’ and other variations of it, to no avail.
If anyone could post a hint of a solution I’d appreciate, thanks.
Advertisement
Answer
#!/bin/bash python3 /home/me/workspace/project/foo.py
I think this is your problem. @Martijn Pieters is spot on when he says it is a path issue. You need to ensure that the ‘python3’ you’re calling is the python3 that has selenium installed. Wherever you’re doing the pip3 freeze from, you need to ensure that is what the cron sees.
When I use cronjobs I have my anaconda virtualenv explicitly run the job, as follows:
/home/username/anaconda/envs/python3/bin/python /home/me/workspace/project/foo.py