Currently there are 2 versions of Python installed on the cluster: Python 2.7 and Python 3.7
A lot of packages are already installed globally.
You can see the list of installed packages and their version with these commands.
To see packages available for Python 2.7 use:
pip list
If you want the output in requirements format
pip freeze
You can use grep to search for a specific package
pip list | grep package-name
pip freeze | grep package-name
To see packages available for Python 3.7 use same commands as for Python 2.7, just use pip3 instead of pip
pip3 list
pip3 freeze
pip3 list | grep package-name
pip3 freeze | grep package-name
In case you need a different version of a certain package or a package that isn’t installed the preferred method is using virtualenv.
virtualenv allows creating an isolated (or semi-isolated) python environment where you can install your own packages without requiring super-user privileges.
virtualenv can take a lot of disk space, so it should be created on a lab storage space and not on the home directory.
To create a virtual environment with Python 3 in a lab's storage use this command:
virtualenv /sci/labs/pi-user-name/your-user-name/my-python-venv --python python3
To create the virtual environment for Python 2.7 use:
virtualenv /sci/labs/pi-user-name/your-user-name/my-python-venv
This command will create a folder with the name my-python-venv.
The Python virtual environment will be created in that folder.
We suggest you give that virtual environment a meaningful name so you would remember the purpose of creating it.
Please avoid copying the example name.
To install packages in the virtualenv you created you first need to activate it:
For csh shell (default shell in cluster):
source /sci/labs/pi-user-name/your-user-name/my-python-venv/bin/activate.csh
For sh, bash or zsh shells
. /sci/labs/pi-user-name/your-user-name/my-python-venv/bin/activate
Notice that there is a dot at the beginning of the command instead of the command “source” as for csh shell.
Now you can install packages inside the virtualenv with pip command
pip install package-name
To install a specific version of the package use:
pip install package-name==version
To uninstall a package:
pip uninstall package-name
Packages that come from a source with a setup.py can also be installed inside the virtualenv
python setup.py install
When the installation of packages is finished you need to exit the virtualenv with this command:
deactivate
To use the packages you installed in virtualenv you need to activate it the same way you activate it for installing packages.
For interactive work in the cluster with csh shell (default shell in cluster):
source /sci/labs/pi-user-name/your-user-name/my-python-venv/bin/activate.csh
For sbatch scripts sh, bash or zsh shells and scripts:
. /sci/labs/pi-user-name/your-user-name/my-python-venv/bin/activate
After activating, any Python related commands will work from the virtualenv.
In scripts use this line at the beginning so that your script will use the Python interpreter from the virtualenv :
#!/usr/bin/env python
You can verify that you are using the Python in the virtualenv with this command:
which python
Output should be
/sci/labs/pi-user-name/your-user-name/my-python-venv/bin/python
When you finish working with the virtualenv deactivate it:
deactivate
To make sure that your Python script will use the first Python interpreter in the PATH you need to use this line at the beginning of your script:
#!/usr/bin/env python
The Python interpreter will change when you load some software with the module command that has Python packages.
To check which Python interpreter you are using run this command:
which python