{
"cells": [
{
"cell_type": "markdown",
"id": "14bcabbb7590d770",
"metadata": {},
"source": [
"# Oxford Flower VLAD and Fisher Vector Retrieval Demo\n",
"\n",
"This notebook demonstrates how to:\n",
"1. Load the Oxford Flower dataset.\n",
"2. Extract deep convolutional features (last conv layer) from the `resnet18` model.\n",
"3. Train a VLAD model on these deep features.\n",
"4. Perform image retrieval queries.\n",
"5. Show the effect of PCA (reducing features by half before VLAD) on retrieval performance.\n",
"6. An analogous procedure is made for Fisher Vectors\n",
"\n",
"### References\n",
"\n",
"[1] Relja Arandjelović and Andrew Zisserman, 'All About VLAD', Department of Engineering Science, University of Oxford. \\\n",
"[2] Liangliang Wang and Deepu Rajan, \"An Image Similarity Descriptor for Classification Tasks,\" J. Vis. Commun.\n",
"Image R., vol. 71, pp. 102847, 2020."
]
},
{
"cell_type": "markdown",
"id": "efe5e6d1733eebf3",
"metadata": {},
"source": [
"\n",
"\n",
"## 1. Imports and Setup"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "169b9b052dc0e9b4",
"metadata": {},
"outputs": [],
"source": [
"import os\n",
"from collections.abc import Sequence\n",
"\n",
"import matplotlib.pyplot as plt\n",
"import numpy as np\n",
"import torch\n",
"from PIL import Image\n",
"\n",
"from pyvisim.classic import FisherVectorEmbedder, VLADEmbedder\n",
"from pyvisim.datasets import OxfordFlowerDataset\n",
"\n",
"# Our library imports\n",
"from pyvisim.neural_networks.features import DeepConvFeature\n",
"from pyvisim.retrieval.data import Candidate\n",
"from pyvisim.retrieval.image_store import InMemoryImageEmbeddingStore\n",
"from pyvisim.typing import UInt8NumpyArray"
]
},
{
"cell_type": "markdown",
"id": "df569f572035be0c",
"metadata": {},
"source": [
"### Hyperparameters\n",
"\n",
"> [!NOTE]\n",
"> - Training k-means models takes quite a bit of time. In this notebook, a single `n_clusters = 32` will be used. Change `NUM_CLUSTERS` to experiment with different cluster sizes.\n",
">\n",
"> - `IMAGE_STEP` keeps every `IMAGE_STEP`-th training image, both to train the embedders and to build the image stores. Set to `1` if you want to use all images."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "ef9a676a290a13c6",
"metadata": {},
"outputs": [],
"source": [
"NUM_CLUSTERS = 32\n",
"DIM_REDUCTION_FACTOR = 2\n",
"IMAGE_STEP = 4"
]
},
{
"cell_type": "markdown",
"id": "551e3694",
"metadata": {},
"source": [
"### Helper functions"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "9c872167",
"metadata": {},
"outputs": [],
"source": [
"def plot_image(image: UInt8NumpyArray | torch.Tensor, title: str = \"Image\") -> None:\n",
" \"\"\"\n",
" Plot a single image.\n",
"\n",
" :param image: Image as a NumPy array (H, W, C) or torch tensor (C, H, W)\n",
" :param title: Title of the plot\n",
" \"\"\"\n",
" plt.figure(figsize=(10, 10))\n",
" if isinstance(image, torch.Tensor):\n",
" image = image.detach().cpu()\n",
" if image.ndim == 3:\n",
" image = image.permute(1, 2, 0)\n",
" image = image.numpy()\n",
" plt.imshow(image)\n",
" plt.axis(\"off\")\n",
" plt.title(title)\n",
" plt.show()\n",
"\n",
"\n",
"def plot_candidates(\n",
" query_image: UInt8NumpyArray,\n",
" query_label: int,\n",
" candidates: Sequence[Candidate],\n",
" labels_by_path: dict[str, int],\n",
") -> None:\n",
" \"\"\"\n",
" Plot a query image next to the gallery images retrieved for it.\n",
"\n",
" :param query_image: Query image as a NumPy array (H, W, C)\n",
" :param query_label: Class label of the query image\n",
" :param candidates: Ranked candidates retrieved for the query, e.g. by\n",
" :meth:`~pyvisim.retrieval.image_store.InMemoryImageEmbeddingStore.retrieve_top_k_similar`\n",
" :param labels_by_path: Class label of every gallery image, keyed by its path\n",
" :raises KeyError: If a candidate path is missing from ``labels_by_path``\n",
" \"\"\"\n",
" num_plots = len(candidates) + 1\n",
" _, axes = plt.subplots(1, num_plots, figsize=(4 * num_plots, 4), squeeze=False)\n",
" axes[0, 0].imshow(query_image)\n",
" axes[0, 0].set_title(f\"Query image. Label: {query_label}\")\n",
" for axis, candidate in zip(axes[0, 1:], candidates, strict=True):\n",
" axis.imshow(_read_rgb_image(candidate.path))\n",
" axis.set_title(\n",
" f\"Retrieved image. Label: {labels_by_path[candidate.path]}\\n\"\n",
" f\"Score: {candidate.score:.4f}\"\n",
" )\n",
" for axis in axes[0]:\n",
" axis.axis(\"off\")\n",
" plt.show()\n",
"\n",
"\n",
"def _read_rgb_image(path: str) -> UInt8NumpyArray:\n",
" \"\"\"\n",
" Read an image file into an RGB array.\n",
"\n",
" :param path: Path to the image file\n",
" :return: The image as a NumPy array (H, W, 3)\n",
" \"\"\"\n",
" with Image.open(path) as image:\n",
" return np.asarray(image.convert(\"RGB\"))"
]
},
{
"cell_type": "markdown",
"id": "b0add8d2ea6e3f52",
"metadata": {},
"source": [
"## 2. Declare the Oxford Flower Dataset"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "8bc666f136bd5e5a",
"metadata": {},
"outputs": [],
"source": [
"train_dataset = OxfordFlowerDataset(purpose=\"train\")\n",
"val_dataset = OxfordFlowerDataset(purpose=\"validation\")\n",
"print(\"Number of images in the dataset:\", len(train_dataset))\n",
"\n",
"train_indices = range(0, len(train_dataset), IMAGE_STEP)\n",
"print(\"Number of images used for training and retrieval:\", len(train_indices))"
]
},
{
"cell_type": "markdown",
"id": "e1b5d70eaedec419",
"metadata": {},
"source": [
"### Plot some images from the dataset"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "b6d3bef8b291a2a",
"metadata": {},
"outputs": [],
"source": [
"for i in range(5):\n",
" img, label, _ = train_dataset[i]\n",
" print(\"Image size:\", img.shape)\n",
" plot_image(img, title=f\"Label: {label}\")"
]
},
{
"cell_type": "markdown",
"id": "8465663d29f27ba8",
"metadata": {},
"source": [
"### 3. Extract deep convolutional features\n",
"\n",
"In the original paper [[1]], `SIFT` and `RootSIFT` features were used. Hence, the default parameter of the embedding would be `RootSIFT`. However, here, I would like to demonstrate the usage of deep convolutional features, as mentioned in [[2]].\n",
"\n",
"We use `DeepConvFeature` from our code. For demonstration, we'll pick `resnet18` and the last conv layer."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "af5d4c798fa02c7d",
"metadata": {},
"outputs": [],
"source": [
"extractor = DeepConvFeature(\n",
" backbone=\"resnet18\",\n",
" layer_index=-1, # Last conv layer\n",
")"
]
},
{
"cell_type": "markdown",
"id": "a82fc50861718f39",
"metadata": {},
"source": [
"### Declare the VLAD embedder"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "934c964a7fab03ea",
"metadata": {},
"outputs": [],
"source": [
"vlad_embedder_no_pca = VLADEmbedder(feature_extractor=extractor, n_clusters=NUM_CLUSTERS)"
]
},
{
"cell_type": "markdown",
"id": "d6d29f4a19dfe5d8",
"metadata": {},
"source": [
"The following cell trains the model from scratch on the training images. It might take quite a bit of time."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "c13441c65b6b86ca",
"metadata": {},
"outputs": [],
"source": [
"vlad_embedder_no_pca.learn(train_dataset[i][0] for i in train_indices)"
]
},
{
"cell_type": "markdown",
"id": "9fcaed5d404fec5d",
"metadata": {},
"source": [
"If you have issue with the runtime, you can follow this procedure instead:\n",
"1) Train the embedder once.\n",
"2) Save it using `save_to_disk`.\n",
"3) Load it back later using `VLADEmbedder.load_from_disk`."
]
},
{
"cell_type": "markdown",
"id": "5e138a27240f6c67",
"metadata": {},
"source": [
"### Build the image store\n",
"\n",
"This embeds the selected training images and indexes the embeddings in an `InMemoryImageEmbeddingStore`. The store will come handy as we do image retrieval later on. For that, we first need to compute some variables from the dataset."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "df024255034f1001",
"metadata": {},
"outputs": [],
"source": [
"paths = [train_dataset.image_paths[i] for i in train_indices]\n",
"labels_by_path = dict(zip(train_dataset.image_paths, train_dataset.labels, strict=True))"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "a9050c20fdfaa7fc",
"metadata": {},
"outputs": [],
"source": [
"vlad_store = InMemoryImageEmbeddingStore(\n",
" image_paths=paths, embedder=vlad_embedder_no_pca\n",
")\n",
"vlad_store.build_store()"
]
},
{
"cell_type": "markdown",
"id": "7ce8a1dce8152a8e",
"metadata": {},
"source": [
"Similar to above, but here, the dimension of each feature vector is reduced `by half` using `PCA`."
]
},
{
"cell_type": "markdown",
"id": "4336e1bc0fd1c7ec",
"metadata": {},
"source": [
"### Declare the VLAD embedder (with PCA)"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "686d61df8f496c78",
"metadata": {},
"outputs": [],
"source": [
"vlad_embedder_with_pca = VLADEmbedder(feature_extractor=extractor, n_clusters=NUM_CLUSTERS)"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "fdd8474987241ca5",
"metadata": {},
"outputs": [],
"source": [
"vlad_embedder_with_pca.learn(\n",
" (train_dataset[i][0] for i in train_indices), dim_reduction_factor=DIM_REDUCTION_FACTOR\n",
")"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "1705d8f9d60520c2",
"metadata": {},
"outputs": [],
"source": [
"vlad_store_pca = InMemoryImageEmbeddingStore(\n",
" image_paths=paths, embedder=vlad_embedder_with_pca\n",
")\n",
"vlad_store_pca.build_store()"
]
},
{
"cell_type": "markdown",
"id": "32a7c59ec470f6a0",
"metadata": {},
"source": [
"## **5. Compare some images**\n",
"\n",
"We will now use the trained VLAD embedders to compute similarity between some images."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "a0869bc2dfce3437",
"metadata": {},
"outputs": [],
"source": [
"image_1, label_1, path_1 = train_dataset[2005]\n",
"image_2, label_2, path_2 = val_dataset[401]\n",
"plot_image(image_1)\n",
"plot_image(image_2)"
]
},
{
"cell_type": "markdown",
"id": "5c07e5482792e87e",
"metadata": {},
"source": [
"Now, we compare the two images. `cosine similarity` is used in this case."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "73232af235817368",
"metadata": {},
"outputs": [],
"source": [
"sim_with_pca = vlad_embedder_with_pca.similarity_score(image_1, image_2)\n",
"print(\"Similarity Score, with PCA: \", sim_with_pca)\n",
"sim_without_pca = vlad_embedder_no_pca.similarity_score(image_1, image_2)\n",
"print(\"Similarity Score, without PCA: \", sim_without_pca)"
]
},
{
"cell_type": "markdown",
"id": "63a835e73d14452a",
"metadata": {},
"source": [
"## **6. Fetch the most similar image in the dataset, given a query image**"
]
},
{
"cell_type": "markdown",
"id": "8b397e0507db1108",
"metadata": {},
"source": [
"\n",
"Now, we will select an image in the validation dataset, on which the model is not yet trained:"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "fd4237c2992924b6",
"metadata": {},
"outputs": [],
"source": [
"query_image, query_label, query_path = val_dataset[103]"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "402ece71",
"metadata": {},
"outputs": [],
"source": [
"plot_image(query_image, title=f\"Query image. Label: {query_label}\")"
]
},
{
"cell_type": "markdown",
"id": "c250e4ffc079fefc",
"metadata": {},
"source": [
"Retrieve top-k most similar images using the stores built above. We will see how it works, with and without PCA.\n",
"\n",
"Each result is a `Candidate` holding the `path` of the retrieved image and its `score`. The stores are built in the `cosine` space, so the score is the cosine distance `1 - cosine_similarity`, and **lower means more similar**."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "76cecc3b34d91f42",
"metadata": {},
"outputs": [],
"source": [
"top_k_vlad_pca = vlad_store_pca.retrieve_top_k_similar(query_image)[0]\n",
"print(\"Evaluation of VLAD with PCA:\")\n",
"for candidate in top_k_vlad_pca:\n",
" print(\n",
" f\"Path: {os.path.basename(candidate.path)}, Cosine distance: {candidate.score:.4f}\"\n",
" )\n",
"\n",
"top_k_vlad_no_pca = vlad_store.retrieve_top_k_similar(query_image)[0]\n",
"print(\"\\nEvaluation of VLAD without PCA:\")\n",
"for candidate in top_k_vlad_no_pca:\n",
" print(\n",
" f\"Path: {os.path.basename(candidate.path)}, Cosine distance: {candidate.score:.4f}\"\n",
" )"
]
},
{
"cell_type": "markdown",
"id": "6c13ad935ca5c08c",
"metadata": {},
"source": [
"### Let's plot the top-k images next to each other."
]
},
{
"cell_type": "markdown",
"id": "798f2a32591afa44",
"metadata": {},
"source": [
"\n",
"a) Using Model trained on data with PCA"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "c34d93f22a771bdc",
"metadata": {},
"outputs": [],
"source": [
"plot_candidates(query_image, query_label, top_k_vlad_pca, labels_by_path)"
]
},
{
"cell_type": "markdown",
"id": "2369f859159edc7e",
"metadata": {},
"source": [
"b) Using model trained on full data"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "d2892ae33554a7d3",
"metadata": {},
"outputs": [],
"source": [
"plot_candidates(query_image, query_label, top_k_vlad_no_pca, labels_by_path)"
]
},
{
"cell_type": "markdown",
"id": "87f3f0d2ee77ea91",
"metadata": {},
"source": [
"## **7. Similar to above, we will do the exact things for the Fisher Vector**\n",
"\n",
"The implementation for both VLAD and Fisher Vectors are identical. After all, VLAD is simply a simplified case of Fisher Vector."
]
},
{
"cell_type": "markdown",
"id": "061e23b5",
"metadata": {},
"source": [
"The Fisher Vectors are about twice as large as the VLAD vectors, so we free the memory held by the VLAD stores before building the Fisher Vector stores."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "c8244995",
"metadata": {},
"outputs": [],
"source": [
"del vlad_store, vlad_store_pca"
]
},
{
"cell_type": "markdown",
"id": "1e6005a492360d12",
"metadata": {},
"source": [
"### Instantiate Fisher Vector Embedder"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "c8da739a419c04ab",
"metadata": {},
"outputs": [],
"source": [
"fisher_embedder_no_pca = FisherVectorEmbedder(\n",
" feature_extractor=extractor, n_components=NUM_CLUSTERS\n",
")"
]
},
{
"cell_type": "markdown",
"id": "82e809cc3ebcb66d",
"metadata": {},
"source": [
"The following cell trains the model from scratch as well. It might take quite a bit of time (even longer than VLAD)."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "9d86ffdbe27b3111",
"metadata": {},
"outputs": [],
"source": [
"fisher_embedder_no_pca.learn(train_dataset[i][0] for i in train_indices)"
]
},
{
"cell_type": "markdown",
"id": "a3d2714bb18cd332",
"metadata": {},
"source": [
"### Build the image store"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "f6f5431815920476",
"metadata": {},
"outputs": [],
"source": [
"fisher_store = InMemoryImageEmbeddingStore(\n",
" image_paths=paths, embedder=fisher_embedder_no_pca\n",
")\n",
"fisher_store.build_store()"
]
},
{
"cell_type": "markdown",
"id": "8a9c9e5e93f477e9",
"metadata": {},
"source": [
"Similar to above, if you run into runtime issues, consider saving the trained embedder using `save_to_disk` and loading it back later. You can also fit a Gaussian Mixture model in advance and pass it to `load_clustering_model_from_sklearn`. The Gaussian Mixture object can be imported as:\n",
"\n",
"```python\n",
"from sklearn.mixture import GaussianMixture\n",
"```\n",
"\n",
"Note that only `covariance_type=\"diag\"` is supported. Otherwise, the implementation is identical to that of `VLADEmbedder`."
]
},
{
"cell_type": "markdown",
"id": "3ed83ef02328b932",
"metadata": {},
"source": [
"### Instantiate Fisher Vector Embedder with PCA"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "b39cbc87ce272130",
"metadata": {},
"outputs": [],
"source": [
"fisher_embedder_with_pca = FisherVectorEmbedder(\n",
" feature_extractor=extractor, n_components=NUM_CLUSTERS\n",
")"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "edc7978aa5eb5e9",
"metadata": {},
"outputs": [],
"source": [
"fisher_embedder_with_pca.learn(\n",
" (train_dataset[i][0] for i in train_indices), dim_reduction_factor=DIM_REDUCTION_FACTOR\n",
")"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "c0d6b2aaf56e95eb",
"metadata": {},
"outputs": [],
"source": [
"fisher_store_pca = InMemoryImageEmbeddingStore(\n",
" image_paths=paths, embedder=fisher_embedder_with_pca\n",
")\n",
"fisher_store_pca.build_store()"
]
},
{
"cell_type": "markdown",
"id": "4afb94829b10f898",
"metadata": {},
"source": [
"### Compute similarity of two images"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "cfff9c3b96ada940",
"metadata": {},
"outputs": [],
"source": [
"image_similarity_with_pca = fisher_embedder_with_pca.similarity_score(image_1, image_2)\n",
"image_similarity_without_pca = fisher_embedder_no_pca.similarity_score(image_1, image_2)\n",
"print(\"Fisher Similarity Score, with PCA: \", image_similarity_with_pca)\n",
"print(\"Fisher Similarity Score, without PCA: \", image_similarity_without_pca)"
]
},
{
"cell_type": "markdown",
"id": "606d35e2622a7168",
"metadata": {},
"source": [
"### Retrieve top-k most similar images"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "659cfb77",
"metadata": {},
"outputs": [],
"source": [
"plot_image(query_image, title=f\"Query image. Label: {query_label}\")"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "45ad5bdd03538dd3",
"metadata": {},
"outputs": [],
"source": [
"top_k_fisher_pca = fisher_store_pca.retrieve_top_k_similar(query_image)[0]\n",
"print(\"Evaluation of Fisher Vector with PCA:\")\n",
"for candidate in top_k_fisher_pca:\n",
" print(\n",
" f\"Path: {os.path.basename(candidate.path)}, Cosine distance: {candidate.score:.4f}\"\n",
" )\n",
"\n",
"top_k_fisher_no_pca = fisher_store.retrieve_top_k_similar(query_image)[0]\n",
"print(\"\\nEvaluation of Fisher Vector without PCA:\")\n",
"for candidate in top_k_fisher_no_pca:\n",
" print(\n",
" f\"Path: {os.path.basename(candidate.path)}, Cosine distance: {candidate.score:.4f}\"\n",
" )"
]
},
{
"cell_type": "markdown",
"id": "40c873bae24ce1a9",
"metadata": {},
"source": [
"### Let's plot top-k images next to each other.\n",
"a) Using Model trained on Data with PCA"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "5e3aca3dbbedb854",
"metadata": {},
"outputs": [],
"source": [
"plot_candidates(query_image, query_label, top_k_fisher_pca, labels_by_path)"
]
},
{
"cell_type": "markdown",
"id": "dc5fd2aa588c610f",
"metadata": {},
"source": [
"b) Using Model trained on full data"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "bd80586a9a8312f8",
"metadata": {},
"outputs": [],
"source": [
"plot_candidates(query_image, query_label, top_k_fisher_no_pca, labels_by_path)"
]
}
],
"metadata": {},
"nbformat": 4,
"nbformat_minor": 5
}