Getting Started with QIIME2 and Jupyter Lab

Working from a Windows OS and connecting via VPN to a remote server

learning journey
Author

Sarah Tanja

Published

August 10, 2023

Sometimes it seems the hardest part about developing a new workflow is deciding where to put it! As I start working on analyzing my microbiome data, I need to have a Linux OS computing environment with conda, qiime2 all running in a jupyter notebook.

After various tries installing in VirtualBox, uninstalling, re-installing, running into big issues with package dependency conflicts, and some strange temporary failure errors, I finally got things setup to run qiime2 commands in a jupyter notebook on a Linux OS remote server (which has more computing power and space than my PC or laptop).

So the rest of this post details those steps; in case I ever (God willing) have to do it again or explain it to someone else.

Note

Run qiime2 commands in a jupyter notebook on a remote Linux server (from a Windows OS!)

Step 1. Get setup as a user on a remote Linux server

This may be the hardest part because you are relying on support from your institution or PI (unless you have the ability to setup your own Linux server, which is outside the scope of this post).

In Spring of 2023 I had taken the Bioinformatics FISH 546 taught by Steven Roberts at UW. In that class, I was added as a user to a remote Linux server hosted by the Roberts Lab. I got permission to continue using the remote server for my Master’s Thesis research bioinformatics analysis. So, the Roberts Lab set me up with a username and a password to access the remote server.

Step 2. Download Windows Subsystem for Linux (WSL) to your Windows OS computer

I followed the instructions found at the Microsoft Documentation website Install Linux on Windows with WSL.

In brief, I opened Windows PowerShell and ran wsl --install which installed the Ubuntu Linux distribution. I kept the default Ubuntu distribution. I continued following the instructions to Set up your Linux user info, and kept my password secret and safe.

Step 3. Download a VPN Client

As a student at UW, I have access to some free software through the IT department, and to connect remotely to servers hosted at UW I needed to download the virtual private network (VPN) Big-IP Edge Client to my home PC and my mobile laptop.

So I followed these instructions to download the BIG-IP Edge Client for UW students, and I did this for both my home PC and laptop. I then connected to the VPN!

Important

When connected to the VPN Client, make sure the Server selected is ‘All Internet Traffic’, otherwise Jupyter Lab local host won’t connect!

Important

You do not need to connect to the VPN Client when you are on the same network as the remote server! The VPN Client is only needed when working from home/cafe/or on travel.

Step 4. Setup a ‘tunnel’ into the remote server

I opened the WSL Ubuntu terminal 🐧 and made a .ssh directory in my home directory, and then opened the nano text editor to write a config file in the .ssh folder:

mkdir .ssh
nano ~/.ssh/config
Tip

The .ssh directory is hidden, and you won’t see it when you run ls

In the nano text editor I copied the following:

    # Jupyter Notebook tunnel
    Host <server>_tunnel
        HostName <123.45.678.90>
        User <username>
        ControlMaster auto
        ServerAliveInterval 30
        ServerAliveCountMax 1200
        ForwardX11 yes
        LocalForward localhost:9000 127.0.0.1:9000

where:

  • <server> is the remote server name (make this short and sweet, since you will execute this as a command later on and must needs type it out)

  • <123.45.678.90> is the remote server IP address in the HostName field

  • <username> is the username setup for the remote server in the User field

I saved the config file by writing it out (Ctrl+X) and pressing Enter.

Next, I tested the tunnel by opening a new Linux Ubuntu terminal and running:

ssh <server>_tunnel

In addition, you can test direct access to the remote server (not using the tunnel) with:

ssh <username>@<IP-address>

The tunnel connected me to the remote server and prompted me to enter the password associated with my user on the remote server.

Tip

Once tunneled into the remote server you’ll notice the <username>@<server> in the terminal changes from your local computer to the remote server.

Step 5. Follow QIIME2 Install

I followed the QIIME2 Install Instructions for a native install on a Linux OS. The steps below are taken directly from the QIIME2 docs instruction on Natively installing QIIME2 using Miniconda on a Linux machine.

Step 5.1 Install Miniconda

Miniconda provides the conda environment and package manager, and is the recommended way to install QIIME 2. Follow the Miniconda instructions for downloading and installing Miniconda.

Here, I choose to work with Miniconda3, because I also have Python 3. I am also following instructions for downloading to a Linux environment.

Important

It is important to follow all of the directions, particularly ensuring that you run conda init at the end of the installation process.

In the remote server terminal run:

curl -O https://repo.anaconda.com/miniconda/Miniconda3-py311_23.5.2-0-Linux-x86_64.sh

Step 5.2 Install QIIME2 within a conda environment

I continued following the the QIIME2 Install Instructions and, in the remote server terminal ran:

wget https://data.qiime2.org/distro/core/qiime2-2023.5-py38-linux-conda.yml
conda env create -n qiime2-2023.5 --file qiime2-2023.5-py38-linux-conda.yml
rm qiime2-2023.5-py38-linux-conda.yml

Step 5.3 Activate conda environment

conda activate qiime2-2023.5

Step 5.4 Test Installation

qiime --help

Step 6. Download Jupyter Lab in QIIME2 conda environment

Now that I am operating on the remote server, and within the qiime2 conda environment (what a feat!) I can download jupyter lab inside the qiime2-2023.5 env.

sarahtanja@MINERVA:~$ # I started here
(base) stanja@raven:~$ # after ssh. into raven_tunnel
(qiime2-2023.5) stanja@raven:~$ # after conda activate

Step 7. Enable qiime2 plugins for Python API use

jupyter serverextension enable --py qiime2 --sys-prefix

Step 7. Review Startup Sequence

If working from a different network than the remote server, start Big IP-Edge VPN Client and connect to ‘All Internet Traffic’ server.

Open Ubuntu Linux terminal

Run ssh <server>_tunnel Enter user pasword Run conda activate qiime2-2023.5 Run jupyter lab --no-browser --port=9000 Ctrl+Click or Copy&Paste the jupyter lab local host url into a browser window

Open a new Python 3 (ipykernel) notebook Run qiime commands either with Python API or with !qiime an exclamation point before bash command line code, Or, like I do, use a combination of both!