In my previous post, I described about featurizing (vectorizing) text with the sentiment analysis example.
In this post, I describe about the featurizing (vectorizing) image using useful sample code in GitHub, and explain how you can use in your real application.

What is “featurizeImage” transformation

The latest Microsoft R Client and Server (version 9.1 and later) with MicrosoftML package includes some new functionalities by the pre-trained models. The new transform named “featurizeImage” is one of these.

This “featurizeImage” vectorizes (featurizes) the image by the model which is trained by the famous ImageNet dataset. As you know, the ImageNet is the huge image dataset including the labels of so many features like the name of objectives with boundary boxes, gestures, species, and other annotations. When using “featurizeImage”, the residual network (ResNet) or convolutional network (AlexNet) can be used for this pre-trained model. (I note that you cannot analyze with the labels or attributes which is not in ImageNet dataset. For example, facial recognition with some specific faces, etc)

You can download the very nice example for this “featurizeImage” transformation from the GitHub (see here), and here I show you how you can use along with this GitHub sample code. (But I changed several lines of code for my example …)

Our Sample

Here we use the following pictures which I’ve taken during my vacations before.
The picture #1 and #5 are the landmarks of ancient Greece. #4 and #7 are the house in shiny Santorini island. #2, #3, and #6 are the animals in the zoological park (in Tokyo) near my home town.

Vectorizing (Featurizing) Images

First, let’s just see the vectorized (featurized) results of these images. (In new MicrosoftML package, you can just transform and see the result with the data frame.)
Please see the following code.

library("MicrosoftML")

orgdat <- data.frame(
  Image = c(
    "C:\tmp\001.JPG",
    "C:\tmp\002.JPG",
    "C:\tmp\003.JPG",
    "C:\tmp\004.JPG",
    "C:\tmp\005.JPG",
    "C:\tmp\006.JPG",
    "C:\tmp\007.JPG"),
  stringsAsFactors = FALSE)

vecdat <- rxFeaturize(
  data = orgdat,
  mlTransforms = list(
    loadImage(vars = list(Features = "Image")),
    resizeImage(vars = "Features", width = 224, height = 224),
    extractPixels(vars = "Features"),
    featurizeImage(var = "Features", dnnModel = "resnet101")   
  ))

As you can see, transformation steps are the followings :

1. loadImage – loading images from each path name (in “Image” column)
2. resizeImage – resizing all images to 244 X 244. Later I explain this background.
3. extractPixels – extracting pixels (vector of number) from the loaded binary images.
4. featurizeImage – vectorizing with pre-trained model. (CNTK wrapper seems to be used in the model evaluation.)

The result (data frame “vecdat”) is having 2048 featurized columns (vectors), and the values are the numeric like following.

In this example we’re using ResNet-101 network for the pre-trained model. In this case, the image size must be 244 X 244.
For the table of the input image size for each available networks, you can see the blog post “Microsoft R blog : Image featurization with a pre-trained deep neural network model“.

Image Classification with unsupervised classifier

You can use this featurized image vector for the various learning. One good example is image classification.

The GitHub example (ImageFeaturizer_TrainAndClassifyImage.R) is labeling images (“Fish”, “Helicopter”, etc) and training.
But here, we now classify images without any labeling or training using k-means. That is, the unsupervised classifying approach. (All you need is the vectorized features previously generated.)

Let’s see the following example.
This code classifies by 3 classes depending on the distance of each image vectors (features).

library("MicrosoftML")

orgdat <- data.frame(
  Image = c(
    "C:\tmp\001.JPG",
    "C:\tmp\002.JPG",
    "C:\tmp\003.JPG",
    "C:\tmp\004.JPG",
    "C:\tmp\005.JPG",
    "C:\tmp\006.JPG",
    "C:\tmp\007.JPG"),
  stringsAsFactors = FALSE)

vecdat <- rxFeaturize(
  data = orgdat,
  mlTransforms = list(
    loadImage(vars = list(Features = "Image")),
    resizeImage(vars = "Features", width = 224, height = 224),
    extractPixels(vars = "Features"),
    featurizeImage(var = "Features", dnnModel = "resnet101")   
  ))

result <- kmeans(vecdat[, -1], 3, nstart = 20)

The result (variable “result”) is as follows.
As you can see, each images are fairly well-classified along with your willings. (class #1 = ancient Greek landmarks, class #2 = the house of Santorini, class #3 = animals)

Let’s classify your albums (your real photos) with this program.
All you have to do is to vectorize your photos with pre-trained modeling, and no labeling (by yourself) is needed !

Image Matching (Image Search)

The GitHub example (ImageFeaturizer_FindMatchingImage.R) is also executing matching (search) for the resembling images.
Let’s follow this sample code.

Before running, we prepare another Parthenon photo for search target as follows.

First, we use “dist” function and calculate the euclidean distance for each other.

library("MicrosoftML")

orgdat <- data.frame(
  Image = c(
    "C:\tmp\001.JPG",
    "C:\tmp\002.JPG",
    "C:\tmp\003.JPG",
    "C:\tmp\004.JPG",
    "C:\tmp\005.JPG",
    "C:\tmp\006.JPG",
    "C:\tmp\007.JPG"),
  stringsAsFactors = FALSE)

vecdat <- rxFeaturize(
  data = orgdat,
  mlTransforms = list(
    loadImage(vars = list(Features = "Image")),
    resizeImage(vars = "Features", width = 224, height = 224),
    extractPixels(vars = "Features"),
    featurizeImage(var = "Features", dnnModel = "resnet101")   
  ))

fnddat <- data.frame(
  Image = c("C:\Users\tsmatsuz\Desktop\searching.JPG"),
  stringsAsFactors = FALSE)

vec2dat <- rxFeaturize(
  data = fnddat,
  mlTransforms = list(
    loadImage(vars = list(Features = "Image")),
    resizeImage(vars = "Features", width = 224, height = 224),
    extractPixels(vars = "Features"),
    featurizeImage(var = "Features", dnnModel = "resnet101")   
  ))

distVals <- dist(
  rbind(vecdat, vec2dat)[,-1],
  "euclidean")

The result (distVals) is like following. This result is including all distance for 8 images (7 original images + 1 search target) each other. :

Here we just want the result of comparison with the search target (“searching.JPG”). That is, we need the result only on 8th row.

i <- attr(distVals, "Size") # value of "Size" must be 8
eucDist <- as.matrix(distVals)[i, -i]

The retrieved result (“eucDist”) is like following :

As you can see, the photo #1 (the following picture) is having the minimum distance with the search target (“searching.JPG”), because these 2 photos are resembling several features.