Skip to main content

Python package management

venv and pip

pip is a terminal command used to install and upgrade Python packages.

PyPI is the main Python package repository. It’s ‘official’, but that doesn’t mean a lot - like most of these open source package repositories, a poor quality or even malicious package can easily be uploaded there, so do your diligence when picking them.

A Python virtual environment (or venv, for short) is a directory you can install a particular Python executable and Python packages into, away from your machine’s default ones. Typically each project/repo you work on should have a different venv, and then you never have to deal with conflicting requirements between projects. When you ‘activate’ a particular venv, then run python or pip, those commands will work with that venv’s Python executable and Python packages.

Basic usage

NOTE: You may need to delete the .bash_aliases file (rm .bash_aliases) from your home directory for pip to work properly within a virtual environment.

Create a venv for your project, called ‘venv’ (make sure you run this in the Terminal):

cd myproject
python3 -m venv venv

(Add ‘venv’ to your .gitignore file, because this shouldn’t be added to your git repo.)

When you work with your project’s packages in a terminal, you’ll want to ‘activate’ your venv:

. venv/bin/activate

You’ll notice the prompt changes to show that the venv is activated: (venv) jovyan@jupyter-lab-davidread-ju-6966d9b9b4-7zvsk:~/myproject$

With the venv activated you can install some packages using pip:

(venv) $ pip install --user pandas

The packages will get installed to your venv, in venv/lib/python3.7/site-packages/.

You can see what packages are installed using ‘pip freeze’:

(venv) $ pip freeze

With the venv activated, if you run a Python script from the terminal, the package will be available to it. For example:

(venv) $ python3 -c 'import pandas; print(pandas); print("It worked")'
<module 'pandas' from '/home/jovyan/myproject/venv/lib/python3.7/site-packages/pandas/'>
It worked

In JupyterLab, to be able to use the venv’s packages (instead of the system packages), see Using a venv in Jupyter

When you commit your code, to ensure reproducibility, you should also commit an up-to-date record of what packages you’ve installed. The simplest way is to do:

(venv) $ pip freeze >requirements.txt
(venv) $ git add requirements.txt

You should also add to your README file the instructions for using requirements.txt - see the following section.

Using a project that has a requirements.txt

If a project has a ‘requirements.txt’ then you should install that into a venv.

A project’s README file is the traditional place to communicate usage of a requirements.txt. Because of that, this section is provided in markdown format so it can be copied into your project’s README, and tailored as necessary:

## Setup

Before you can run this project, you need to install some Python packages using the terminal:

# create a virtual environment
cd myproject
python3 -m venv venv

# install the python packages required
. venv/bin/activate
pip install -r requirements.txt

Library conflicts & warnings

If you come across any conflicts or warnings when installing your libraries using pip we advise you use poetry to resolve them.

This page was last reviewed on 1 May 2022. It needs to be reviewed again on 1 July 2022 by the page owner #analytical-platform-support .
This page was set to be reviewed before 1 July 2022 by the page owner #analytical-platform-support. This might mean the content is out of date.