Classifier to Segmentation

We aim to solve the problem of object segmentation, where the input is an image and class label such as “cat” and the output is a mask where objects of that class are present in the image. However, we want to only use a pre-trained classification model with no additional training for segmentation, by checking which pixels of the image were used by the model to classify it as a particular label.

Current state of the art object segmentation methods involve training a deep neural network on a segmentation dataset. DeconvNet in particular appends deconvolutional layers after a convolutional neural network that aim to reverse the convolutions to obtain the input pixels that contributed to a class. However, Simonyan, et. al. in their paper “Deep inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps” proposed a method for doing this same computation with only a pre-trained classification model and no additional training for segmentation. They additionally proved that their method is equivalent to deconvolutions except when dealing with a ReLU layer. They use the gradient of a class probability with respect to an input pixel as a proxy for how important the pixel is for identifying that class, creating a saliency map, which can then be converted into a segmentation mask. A second method, guided backpropagation, tries to improve upon Simonyan, et. al.’s work. We replicate Simonyan, et. al.’s method and compare it in detail to guided backpropagation on the COCO segmentation challenge. We were able to achieve high IOU compared to ground truth masks on images with one large object, but lower IOU on images with many or small objects.

Pipeline Summary

1. Get detected classes with TResNet (multi-class classifier trained on COCO. Input: image. Output: probabilities for each class)

  a. Run the classification model forward on image to get class probabilities

  b. Get top k=20 most likely classes

  c. Filter for classes with probability > threshold=0.75

2. For each detected class:

  a. Calculate the gradient of the class probability w.r.t. the input image

    i. Method 1: “vanilla” gradient backpropagation

    ii. Method 2: guided backpropagation

  b. Convert the gradient to a saliency map by taking the absolute value of the gradient and taking the max absolute value across     the channels for each pixel

  c. Convert the saliency map to a segmentation mask

    i. Method 1: using the GraphCut library (

      1. Compute the foreground seed, a mask of pixels with saliency above the 95th percentile, and the background seed, a             mask of pixels saliency below the 30th percentile

Foreground (left) and background (right) from vanilla gradient. Top: ‘person’. Bottom: ‘giraffe’.

Foreground (left) and background (right) from guided backpropagation. Top: ‘person’. Bottom: ‘giraffe’.

      2. Input the image, foreground seed, and background seed into the GraphCut library to get a foreground mask

    ii. Method 2: our own recursive expansion algorithm (see below)

"Vanilla" Gradient Backpropagation

This method is called “vanilla” because the idea behind it is simple. Simonyan, et. al. proposed that if the gradient of a class’s probability with regard to an input pixel is large, then that means changing the pixel would cause a large change in the class’ probability, so it must be important for identifying that class. Thus the absolute value of the gradient of a class’s probability with regard to all the pixels in an image is a saliency map for that class. We can get this gradient easily using PyTorch Autograd by calling backward() on the model output, which populates the gradient of the output with regard to each layer, and then getting the gradient of the input image.

Guided Backpropagation

Guided backpropagation aims to solve some of the shortcomings of the vanilla method, where the saliency map is fuzzy and pixels that negatively correlate with the class still have high saliency. We backpropagate the same way, except for a ReLU layer we zero out gradient entries that are either negative or whose pixel was zeroed out in the ReLU forward pass. Intuitively we want to zero out negative gradient pixels because they negatively correlate with the output class. Simonyan, et. al. showed that vanilla gradient backpropagation is equivalent to deconvolutions, except when backpropagating through a ReLU layer. Guided backpropagation’s treatment of ReLU layers is closer to that of deconvolutions.

Gradient to Saliency Map

We used the method proposed by Simonyan et. al., which is to take the absolute value of the gradient and flatten the channels by taking the max absolute value across channels for each pixel. The absolute value is the magnitude of the gradient, or, how important it is for the classification. We also explored not taking the absolute value, but this resulted in masks with holes, as shown in the figures below. The figures contain, from left to right: input image, saliency map, mask, masked image. Detected class: “bird.”

Saliency Map to Segmentation Mask

Using the saliency map of the given image, the program extracts pixels with an intensity that is at the bottom 30% to and pixels at or above the top 90% to create a background and foreground image respectively. Then by using the foreground image, we iterate through the top 50 pixels with the highest saliency and recursively expand outwards to pixels above a certain threshold in an attempt to create a mask. This approach generated an inaccurate mask for some images and for others nothing was generated. To fix this, we used a library that implemented the Graph Cut algorithm to segment the image. This implementation of graph cut works by taking the strokes of the seed image that mark foreground and background and perform a cut on the image. The cut is performed by choosing the cut with the least cumulative weight, where weight of an edge is based on the inverse squared difference between neighboring pixels, such that pixels (pixels in the graph are represented by the nodes) with large differences have low weight and vice versa. After the cuts are created, the mask is output separating the foreground and background.

For evaluation, we combined the segmentation masks for all detected classes within an image into a single mask for that image. Then, we compared our generated mask for an image to its ground truth mask. We calculated how well our generated masks were by calculating the Intersection over Union value between the generated masks and their respective ground truth masks. An Intersection over Union value is the amount of intersecting area between the two masks divided by the union of the two masks’ area. Typically, having an IoU value greater than 0.5 is considered good.