I am trying to install a new kernel on a jupyter environment from bash script.
In a dockerfile that uses jupyter/minimal-notebook as the parent image, I run a bash script that:
- generates a new virtual env:
python -m venv new_kernel
- installs numpy and ipykernel
new_kernel/bin/pip install ipykernel numpy
- registers the kernel:
new_kernel/bin/python -m ipykernel install --name new_kernel --display-name "Python (new_kernel)".
Unfortunately, as a result I get:
- the new label “Python (new_kernel)” available among the kernels, BUT if I use that kernel it does not have numpy installed
- investigating in the new_kernel/lib/python3.9/site-packages folder I find numpy installed. This makes me think that the kernel is not being seen somehow.
How can I get the new kernel working along with the libraries installed?
I understood how it works, so I answer myself, to the benefit of those who will have the same problem.
The environment is loaded fine. Brutally import your new kernel libraries and you should have no problems.
As far as I understand, the cell bash emulator acts on a different and independent context from the python context that is loaded into the notebook. So best to use
list(pkg_resources.working_set) to inspect the environment.
Therefore you can install a new kernel starting with a script like this
python -m venv new_kernel new_kernel/bin/pip install ipykernel numpy new_kernel/bin/python -m ipykernel install --name new_kernel --display-name "Python (new_kernel)"
Reproducing the case
python -m venv new_kernel # a new new_kernel folder has been created, containing the environment # install ipykernel and numpy in the new env new_kernel/bin/pip install ipykernel numpy # install the new kernel tied to the new env new_kernel/bin/python -m ipykernel install --name new_kernel --display-name "Python (new_kernel)" # launch jupyter lab jupyter lab
From the jupyterlab GUI you should see new_kernel among the available kernels. Click it in order to open a new notebook with it already loaded.
Let’s inspect the environment.
If you type
!which pythonin a new cell we get a different path to the new environment.
If you type
!pip list, we do not see numpy installed
- By typing the following instructions we find that the environment path has been correctly loaded
import sys print(sys.prefix) # should see something like /home/jovyan/new_kernel print(sys.path)
- If you inspect the packages loaded in this notebook you’ll find numpy installed!
import pkg_resources list(pkg_resources.working_set)
What is happening?
We have just seen that if we run commands from the bash emulator (
!<command>) of the cell the results are lies; whereas if we inspect the python context of the working kernel we find that the environment has been correctly loaded.
Unfortunately, I have found no official references to this phenomenon so far, and I invite anyone to contribute to this question.
My personal explanation is that the bash emulator accesses the same CLI that launched
jupyter lab, which can be in any path and with any other environment on/off; while the notebook launched by jupyter lab actually loads the context from the kernel configuration.
So they act on two different and independent contexts, so it is better to inspect the python context directly.
Interestingly, this problem is not reproduced if you use nb_conda_kernels, since it uses a custom KernelSpecManager.