9. Advanced examples: cellpose and napari¶

After having presented the basic tools needed to do image processing in Python, we explore two additional topics here: usage of an advanced package for segmentation, and multi-dimensional viewing and exploration. In the first part we learn how to use the package cellpose for cell segmentation, insisting in particular on how one proceeds to use a new software. In the second part, we explore napari an interactive multi-dimensional viewer that can easily be extended via your own plugins.

cellpose¶

In the previous chapters we have seen a classic solution for segmentation. Those tools stay entirely relevant as very often one needs them as a complement even when using more advanced methods. Here we show an example of such an advanced method with cellpose. Cellpose is a generalistic cell segmentation tool based on Deep Learning. It is generalistic because it has been trained on a vast amount of different types of images and is therefore very versatile. So much that it can also be used outside the specific field of bioimaging. Let’s see how to proceed.

Installation¶

When you work on a project, you might search for packages to solve a specific project. Once you have found one you will need to first install it. You usually find installation instruction in the GitHub repository of the package or its documentation. For example here, you can find some instructions in the Readme.

We strongly recommend that you work within an environment for your project, i.e. a “closed” space that contains all the package needed for a given project. This avoids version issues when you start mixing various complex software with different dependencies. Such an environment can typically be created with conda (miniconda with a command line tool, Anaconda with a user interface). If you use such an environment then you can safely install new software for testing, as you can easily start over with a fresh environment is something goes bad.

So now you have discovered cellpose and need to install it. Following the instructions, you have found that you can obtain the software from pypi. You can either install form the command line or directly in your notebook:

pip install cellpose

Collecting cellpose
  Downloading cellpose-1.0.2-py3-none-any.whl (174 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 174.4/174.4 KB 4.3 MB/s eta 0:00:00
?25hRequirement already satisfied: tqdm in /Users/gw18g940/mambaforge/envs/improc_beginner/lib/python3.9/site-packages (from cellpose) (4.63.0)
Collecting numba
  Downloading numba-0.55.1-cp39-cp39-macosx_10_14_x86_64.whl (2.3 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.3/2.3 MB 9.6 MB/s eta 0:00:00:00:0100:01
?25hRequirement already satisfied: tifffile in /Users/gw18g940/mambaforge/envs/improc_beginner/lib/python3.9/site-packages (from cellpose) (2022.2.9)
Requirement already satisfied: natsort in /Users/gw18g940/mambaforge/envs/improc_beginner/lib/python3.9/site-packages (from cellpose) (8.1.0)
Collecting torch>=1.6
  Downloading torch-1.11.0-cp39-none-macosx_10_9_x86_64.whl (129.9 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 129.9/129.9 MB 3.1 MB/s eta 0:00:0000:0100:01
?25hCollecting opencv-python-headless
  Downloading opencv_python_headless-4.5.5.64-cp36-abi3-macosx_10_15_x86_64.whl (46.3 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 46.3/46.3 MB 3.6 MB/s eta 0:00:0000:0100:01
?25hRequirement already satisfied: numpy>=1.20.0 in /Users/gw18g940/mambaforge/envs/improc_beginner/lib/python3.9/site-packages (from cellpose) (1.22.2)
Collecting fastremap
  Downloading fastremap-1.12.2-cp39-cp39-macosx_10_9_x86_64.whl (632 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 632.4/632.4 KB 5.8 MB/s eta 0:00:0000:0100:01
?25hRequirement already satisfied: scipy in /Users/gw18g940/mambaforge/envs/improc_beginner/lib/python3.9/site-packages (from cellpose) (1.8.0)
Requirement already satisfied: typing-extensions in /Users/gw18g940/mambaforge/envs/improc_beginner/lib/python3.9/site-packages (from torch>=1.6->cellpose) (4.1.1)
Collecting llvmlite<0.39,>=0.38.0rc1
  Downloading llvmlite-0.38.0-cp39-cp39-macosx_10_9_x86_64.whl (25.5 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 25.5/25.5 MB 4.0 MB/s eta 0:00:0000:0100:01
?25hRequirement already satisfied: setuptools in /Users/gw18g940/mambaforge/envs/improc_beginner/lib/python3.9/site-packages (from numba->cellpose) (60.9.3)
Collecting numpy>=1.20.0
  Downloading numpy-1.21.5-cp39-cp39-macosx_10_9_x86_64.whl (17.0 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 17.0/17.0 MB 2.1 MB/s eta 0:00:0000:0100:01
?25hInstalling collected packages: torch, numpy, llvmlite, opencv-python-headless, numba, fastremap, cellpose
  Attempting uninstall: numpy
    Found existing installation: numpy 1.22.2
    Uninstalling numpy-1.22.2:
      Successfully uninstalled numpy-1.22.2
Successfully installed cellpose-1.0.2 fastremap-1.12.2 llvmlite-0.38.0 numba-0.55.1 numpy-1.21.5 opencv-python-headless-4.5.5.64 torch-1.11.0
Note: you may need to restart the kernel to use updated packages.

Here we see that the installation was successful.

Using the software¶

Now we can go the documentation and search for an example. Cellpose offers an example to run within a notebook. First we need to import the right part of the package:

from cellpose import models

The next step is to use the segmentation model. In the example we see model = models.Cellpose(gpu=False, model_type='cyto'). The question is: does this work in our case? What does cyto mean? To understand these options, you will have to read a bit the documentation or the API (the list of all possible functions of the package to which you have access). In this case the models are explained in the documentation.

Let us remember what type of image we have by plotting one:

import skimage.io
import matplotlib.pyplot as plt

image = skimage.io.imread('images/19838_1252_F8_1.tif')

image.shape

(2048, 2048, 3)

The image is relatively large and it could take a long time to analyze it on a regular computer with a DL method. So we first downscale it:

image = image[::4, :: 4, :]

fig, ax = plt.subplots(figsize=(10,10))
ax.imshow(image);

So we have a marker for the nucleus (blue) and one for the cytosplasm (red). From the documentation, we understand that in that case we can use the cyto model and use the blue nucleus channel as a “help” for cell segmentation.

model = models.Cellpose(model_type='cyto')

To use the model properly, we have to specify channels like this channels=[1,3] where 1 stand for the red channel (RGB) and 3 for the blue one (RGB).

Let’s try now by copying the command:

channels=[[1,3]]
masks, flows, styles, diams = model.eval(image, diameter=None, channels=channels)

Let’s look at the mask superposed on the image:

fig, ax = plt.subplots(figsize=(10,10))
ax.imshow(image[:,:,0], cmap='gray')
ax.imshow(masks, alpha=0.5);

Not bad for a first trial! Even touching cells get properly segmented which could be difficult to achieve with classical methods. Now we could refine this result by further reading the documentation and seeing that we could provide a diameter estimate:

masks, flows, styles, diams = model.eval(image, diameter=100, channels=channels)

fig, ax = plt.subplots(figsize=(10,10))
ax.imshow(image[:,:,0], cmap='gray')
ax.imshow(masks, alpha=0.5);

We could also segment the nuclei using another model type:

model = models.Cellpose(model_type='nuclei')

channels = [3,0]
masks, flows, styles, diams = model.eval(image, diameter=100, channels=channels)

fig, ax = plt.subplots(figsize=(10,10))
ax.imshow(image[:,:,2], cmap='gray')
ax.imshow(masks, alpha=0.5);

napari¶

To display images, we have until now only used Matplotlib. It allows to have a quick look at an image or to create highgly customized figures for publications but it is limited in interactivity, in particular to visualize multi-dimensional data. napari has been created to fill this void: it provides in the Python eco-system a multi-dimensional viewer which is interactive and highly customizable via a plugin system.

napari can be installed like any other package via pip or conda, but you can also download it as a standalone application. Once installed you can use it as an application and use an editor integrated to it or you can run the software from a notebook. Note that napari doesn’t display in the notebook but in a separate figure, making it unusable in Google Colab.

Let’s first import the package:

import napari
from napari.utils import nbscreenshot

The first thing we have to do is to create the viewer. Once we have a viewer, we can interact with it both via the interface and programmatically via methods attached to it. Below we see the napari viewer: an area on the right where images are viewed, and an interactive menu on the left.

viewer = napari.Viewer()

nbscreenshot(viewer)

Now we can add an image to the viewer. We use here options to select a colormap and to give a name to each channel:

viewer.add_image(image[:,:,0], colormap='red', name='cytoplasm')
viewer.add_image(image[:,:,1], colormap='green', name='signal')
viewer.add_image(image[:,:,2], colormap='blue', name='nuclei');

nbscreenshot(viewer)

We see now a list of layers appearing on the left. napari works with such layers that can be superposed and which can be of different types. The type of layer can be selected when adding data. For example we can add labels now:

viewer.add_labels(masks);

nbscreenshot(viewer)

Each layer type has its own options. For example here in the labels layer, we can choose the currently selected label, set opacity etc. We also have a set of tools to interactively work on the layer. By selecting the pencil or the eraser, we can edit the labels:

nbscreenshot(viewer)

This was just a very brief insight into this very useful software. To learn more, visit the various tutorials on the napari web-site.

napari plugins¶

napari is integrated with a very efficient plugin system. You can find a list of plugins on the napari-hub and install them directly from the interface (menu Plugin). For example you can find an animation plugin to create animations in your viewer:

nbscreenshot(viewer)

You can also very easily add interactivity to napari yourself. For example let’s imagine that you’d like to have a little GUI to do some thresholding. For this you write your thresholding function, add some annotations to it and then you can use a package called magicgui to turn the function into an interactive widget:

import skimage
from magicgui import magicgui
from napari.types import ImageData, LabelsData

def my_thresholder(image: ImageData, th_level: int=0) -> LabelsData: 
    
    label_image = image > th_level
    label_image = skimage.morphology.label(label_image)
    
    return(label_image)

thresholding_widget = magicgui(my_thresholder)

Now we can add the widget to the viewer:

viewer.window.add_dock_widget(thresholding_widget, area='right')

<napari._qt.widgets.qt_viewer_dock_widget.QtViewerDockWidget at 0x1ed4768b0>

nbscreenshot(viewer)

And we can run the function:

nbscreenshot(viewer)

Image processing with Python for beginners

9. Advanced examples: cellpose and napari

Contents