How to set up a new environment and make it available as a kernel in Jupyter Notebook?

Create the YAML file

A YAML file is a human-readable data serialization format often used for configuration files and data exchange between applications. A sample YAML file looks like below. Let’s save it as environment.yaml.

name: testenv
channels:
  - defaults
  - nvidia
  - pytorch 
  - conda-forge
dependencies:
  - pip==21.0.1
  - python==3.8.2
  - pytorch==2.0.0
  - pytorch-cuda==11.8
  - pip:
    - transformers==4.40.2
    - accelerate>=0.21.0

In YAML files, channels are used to specify where packages should be downloaded from. The default channel is provided by Anaconda. It includes a wide range of packages that are commonly used. nvidia channel is maintained by NVIDIA and includes packages related to GPU computing, such as CUDA and cuDNN, which are essential for running deep learning models on NVIDIA GPUs.

We can see that within the dependencies section, there is also a pip subsection. The dependencies section lists packages that Conda will install. These packages are typically available from the channels specified in the YAML file (like defaults, conda-forge, etc.) The pip subsection is used to specify packages that should be installed using pip, the Python package installer. This is useful for packages that are not available through Conda channels.

Create the new environment from the YAML file

When you create an environment from the YAML file, Conda will first install the packages listed under the dependencies section using Conda, and then it will use pip to install the packages listed under the pip subsection.

conda env create -f environment.yaml

# To remove the created environment later
conda remove --name testenv --all

This will create a new Conda environment with the dependencies specified in our YAML file.

Create a Jupyter kernel

After the environment is created, activate it

conda activate testenv

# To deactivate later
conda deactivate

Install the IPython kernel package if it’s not already installed

pip install jupyter ipykernel

Create a new Jupyter kernel for this environment

python -m ipykernel install --user --name testenv --display-name "Python (testenv)"

Use the new kernel in Jupyter Notebook

Now, let’s open or create a Jupyter Notebook on Azure. We can got the Kernel menu and select Change kernel. Choose the new kernel we just created (e.g. Python (testenv))