4.1 Image Search¶
This notebook builds a searchable image gallery and queries it with an image rather than with text. The gallery is the Oxford 102 Flowers dataset [1], each image is turned into a feature vector by a pre-trained CLIP model [2], and the vectors are indexed in an HNSW graph [3] so that the nearest neighbors of a query are found without scanning the gallery.
The pipeline has four steps:
Embed: every gallery image is read and passed through
ClipEmbedder, which returns one vector per image.Index: the vectors are handed to the search index that
InMemoryImageEmbeddingStoreowns from then on.Serialization: the store is written to a single
safetensorsfile, embedder included, so the gallery never has to be embedded twice.Query: an unseen image is embedded with the same embedder and matched against the index.
Note
Note that embedding the whole gallery takes a while. Once the file is on disk, the notebook can be restarted from the loading cell.
Hnsw Search Index¶
A brute-force search compares the query against every gallery vector, which is exact but linear in the size of the gallery. HNSW instead builds a multi-layer proximity graph over the vectors and walks it greedily, which reaches the neighborhood of the query in far fewer comparisons. The trade is recall: the answer is approximate, and how approximate is controlled by the parameters set below.
Further reading: Image Store and External Indexes.
Import libraries¶
import matplotlib.pyplot as plt
import numpy as np
from PIL import Image
from pyvisim.datasets import OxfordFlowerDataset
from pyvisim.neural_networks import ClipEmbedder
from pyvisim.retrieval.image_store import InMemoryImageEmbeddingStore
/home/runner/work/Python-Visual-Similarity/Python-Visual-Similarity/.venv/lib/python3.10/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html
from .autonotebook import tqdm as notebook_tqdm
Hyperparameters¶
Note
IMAGE_STEP keeps every IMAGE_STEP-th image of the training split in the gallery. Set to 1 if you want to use all images.
IMAGE_STEP = 4
Build the image store¶
Constructing InMemoryImageEmbeddingStore only records the gallery. build_store() reads every path in image_paths, embeds it with embedder and hands the resulting matrix to the search index, so call it right after the constructor. Until then the store holds nothing, and searching it raises RuntimeError. Pass lazy_build=False to have the constructor call build_store() itself.
The gallery is every IMAGE_STEP-th image of the training split of Oxford 102 Flowers [1], which holds 6149 images.
Arguments:
embedder: any object exposing anembedmethod.ClipEmbedderis used here, but theVLAD,Fisher Vector,SiameseandTripletembedders fit here just as well.search_index:"hnsw"builds the graph,Nonefalls back to an exact brute-force scan. AnExternalSearchIndexcan be passed instead to search through aFAISSindex [5].index_params: forwarded to the index constructor.graph_degreeis the number of bidirectional links created per node of the graph,build_candidatesthe size of the candidate list kept while building it. Both raise recall,graph_degreeat the cost of memory andbuild_candidatesat the cost of build time.
The metric space defaults to "cosine", which stores the vectors L2-normalised and scores them by 1 - cosine_similarity.
train_dataset = OxfordFlowerDataset()
train_indices = range(0, len(train_dataset), IMAGE_STEP)
train_image_paths = [train_dataset.image_paths[i] for i in train_indices]
print("Number of images in the gallery:", len(train_image_paths))
embedder = ClipEmbedder()
image_store = InMemoryImageEmbeddingStore(
image_paths=train_image_paths,
embedder=embedder,
search_index="hnsw",
index_params={"graph_degree": 16, "build_candidates": 200},
)
image_store.build_store()
image_store.save_to_disk("flower_image_store.safetensors")
Downloading labels.mat: 0%| | 0.00/502 [00:00<?, ?B/s]
Downloading labels.mat: 100%|██████████| 502/502 [00:00<00:00, 2.32MB/s]
Downloading setid.mat: 0%| | 0.00/15.0k [00:00<?, ?B/s]
Downloading setid.mat: 100%|██████████| 15.0k/15.0k [00:00<00:00, 57.6MB/s]
Downloading images.tgz: 0%| | 0.00/345M [00:00<?, ?B/s]
Downloading images.tgz: 0%| | 32.8k/345M [00:00<21:46, 264kB/s]
Downloading images.tgz: 0%| | 98.3k/345M [00:00<14:15, 403kB/s]
Downloading images.tgz: 0%| | 147k/345M [00:00<14:21, 400kB/s]
Downloading images.tgz: 0%| | 246k/345M [00:00<10:31, 546kB/s]
Downloading images.tgz: 0%| | 377k/345M [00:00<07:59, 718kB/s]
Downloading images.tgz: 0%| | 557k/345M [00:00<06:01, 952kB/s]
Downloading images.tgz: 0%| | 803k/345M [00:00<04:31, 1.27MB/s]
Downloading images.tgz: 0%| | 1.15M/345M [00:01<03:20, 1.72MB/s]
Downloading images.tgz: 0%| | 1.62M/345M [00:01<02:26, 2.34MB/s]
Downloading images.tgz: 1%| | 2.29M/345M [00:01<01:45, 3.23MB/s]
Downloading images.tgz: 1%| | 3.23M/345M [00:01<01:16, 4.47MB/s]
Downloading images.tgz: 1%|▏ | 4.52M/345M [00:01<00:55, 6.18MB/s]
Downloading images.tgz: 2%|▏ | 6.31M/345M [00:01<00:39, 8.52MB/s]
Downloading images.tgz: 3%|▎ | 8.86M/345M [00:01<00:28, 12.0MB/s]
Downloading images.tgz: 4%|▎ | 12.3M/345M [00:01<00:20, 16.4MB/s]
Downloading images.tgz: 5%|▍ | 15.9M/345M [00:02<00:16, 19.8MB/s]
Downloading images.tgz: 6%|▌ | 20.0M/345M [00:02<00:14, 23.0MB/s]
Downloading images.tgz: 7%|▋ | 24.0M/345M [00:02<00:12, 25.1MB/s]
Downloading images.tgz: 8%|▊ | 28.0M/345M [00:02<00:11, 26.6MB/s]
Downloading images.tgz: 9%|▉ | 32.1M/345M [00:02<00:11, 27.6MB/s]
Downloading images.tgz: 10%|█ | 36.1M/345M [00:02<00:10, 28.4MB/s]
Downloading images.tgz: 12%|█▏ | 40.1M/345M [00:02<00:10, 28.9MB/s]
Downloading images.tgz: 13%|█▎ | 44.1M/345M [00:02<00:10, 29.0MB/s]
Downloading images.tgz: 14%|█▍ | 48.1M/345M [00:03<00:10, 29.2MB/s]
Downloading images.tgz: 15%|█▌ | 52.1M/345M [00:03<00:09, 29.4MB/s]
Downloading images.tgz: 16%|█▋ | 56.1M/345M [00:03<00:09, 29.5MB/s]
Downloading images.tgz: 17%|█▋ | 60.1M/345M [00:03<00:09, 29.6MB/s]
Downloading images.tgz: 19%|█▊ | 64.1M/345M [00:03<00:09, 29.8MB/s]
Downloading images.tgz: 20%|█▉ | 68.1M/345M [00:03<00:09, 29.7MB/s]
Downloading images.tgz: 21%|██ | 72.1M/345M [00:03<00:09, 29.8MB/s]
Downloading images.tgz: 22%|██▏ | 76.1M/345M [00:04<00:09, 29.8MB/s]
Downloading images.tgz: 23%|██▎ | 80.2M/345M [00:04<00:08, 29.9MB/s]
Downloading images.tgz: 24%|██▍ | 84.1M/345M [00:04<00:08, 29.8MB/s]
Downloading images.tgz: 26%|██▌ | 88.2M/345M [00:04<00:08, 29.8MB/s]
Downloading images.tgz: 27%|██▋ | 92.2M/345M [00:04<00:08, 29.8MB/s]
Downloading images.tgz: 28%|██▊ | 96.2M/345M [00:04<00:08, 29.8MB/s]
Downloading images.tgz: 29%|██▉ | 100M/345M [00:04<00:08, 29.8MB/s]
Downloading images.tgz: 30%|███ | 104M/345M [00:05<00:08, 29.9MB/s]
Downloading images.tgz: 31%|███▏ | 108M/345M [00:05<00:07, 29.9MB/s]
Downloading images.tgz: 33%|███▎ | 112M/345M [00:05<00:07, 29.9MB/s]
Downloading images.tgz: 34%|███▎ | 116M/345M [00:05<00:07, 29.9MB/s]
Downloading images.tgz: 35%|███▍ | 120M/345M [00:05<00:07, 30.0MB/s]
Downloading images.tgz: 36%|███▌ | 124M/345M [00:05<00:07, 30.1MB/s]
Downloading images.tgz: 37%|███▋ | 128M/345M [00:05<00:07, 30.0MB/s]
Downloading images.tgz: 38%|███▊ | 132M/345M [00:05<00:07, 30.0MB/s]
Downloading images.tgz: 40%|███▉ | 136M/345M [00:06<00:06, 29.8MB/s]
Downloading images.tgz: 41%|████ | 140M/345M [00:06<00:06, 29.9MB/s]
Downloading images.tgz: 42%|████▏ | 144M/345M [00:06<00:06, 29.9MB/s]
Downloading images.tgz: 43%|████▎ | 148M/345M [00:06<00:06, 29.9MB/s]
Downloading images.tgz: 44%|████▍ | 152M/345M [00:06<00:06, 30.0MB/s]
Downloading images.tgz: 45%|████▌ | 156M/345M [00:06<00:06, 30.1MB/s]
Downloading images.tgz: 47%|████▋ | 160M/345M [00:06<00:06, 30.1MB/s]
Downloading images.tgz: 48%|████▊ | 164M/345M [00:07<00:06, 30.0MB/s]
Downloading images.tgz: 49%|████▉ | 168M/345M [00:07<00:05, 30.0MB/s]
Downloading images.tgz: 50%|█████ | 172M/345M [00:07<00:05, 29.9MB/s]
Downloading images.tgz: 51%|█████ | 177M/345M [00:07<00:05, 30.0MB/s]
Downloading images.tgz: 52%|█████▏ | 181M/345M [00:07<00:05, 30.0MB/s]
Downloading images.tgz: 54%|█████▎ | 185M/345M [00:07<00:05, 30.0MB/s]
Downloading images.tgz: 55%|█████▍ | 189M/345M [00:07<00:05, 29.9MB/s]
Downloading images.tgz: 56%|█████▌ | 193M/345M [00:07<00:05, 29.9MB/s]
Downloading images.tgz: 57%|█████▋ | 197M/345M [00:08<00:04, 30.1MB/s]
Downloading images.tgz: 58%|█████▊ | 201M/345M [00:08<00:04, 30.0MB/s]
Downloading images.tgz: 59%|█████▉ | 205M/345M [00:08<00:04, 30.0MB/s]
Downloading images.tgz: 61%|██████ | 209M/345M [00:08<00:04, 29.9MB/s]
Downloading images.tgz: 62%|██████▏ | 213M/345M [00:08<00:04, 29.9MB/s]
Downloading images.tgz: 63%|██████▎ | 217M/345M [00:08<00:04, 29.9MB/s]
Downloading images.tgz: 64%|██████▍ | 221M/345M [00:08<00:04, 30.0MB/s]
Downloading images.tgz: 65%|██████▌ | 225M/345M [00:09<00:04, 29.6MB/s]
Downloading images.tgz: 66%|██████▋ | 229M/345M [00:09<00:03, 29.7MB/s]
Downloading images.tgz: 67%|██████▋ | 233M/345M [00:09<00:03, 29.8MB/s]
Downloading images.tgz: 69%|██████▊ | 237M/345M [00:09<00:03, 29.8MB/s]
Downloading images.tgz: 70%|██████▉ | 241M/345M [00:09<00:03, 29.8MB/s]
Downloading images.tgz: 71%|███████ | 245M/345M [00:09<00:03, 29.8MB/s]
Downloading images.tgz: 72%|███████▏ | 249M/345M [00:09<00:03, 29.9MB/s]
Downloading images.tgz: 73%|███████▎ | 253M/345M [00:09<00:03, 30.0MB/s]
Downloading images.tgz: 74%|███████▍ | 257M/345M [00:10<00:02, 30.0MB/s]
Downloading images.tgz: 76%|███████▌ | 261M/345M [00:10<00:02, 30.0MB/s]
Downloading images.tgz: 77%|███████▋ | 265M/345M [00:10<00:02, 29.9MB/s]
Downloading images.tgz: 78%|███████▊ | 269M/345M [00:10<00:02, 29.8MB/s]
Downloading images.tgz: 79%|███████▉ | 273M/345M [00:10<00:02, 29.8MB/s]
Downloading images.tgz: 80%|████████ | 277M/345M [00:10<00:02, 29.9MB/s]
Downloading images.tgz: 81%|████████▏ | 281M/345M [00:10<00:02, 29.9MB/s]
Downloading images.tgz: 83%|████████▎ | 285M/345M [00:11<00:02, 30.0MB/s]
Downloading images.tgz: 84%|████████▍ | 289M/345M [00:11<00:01, 30.0MB/s]
Downloading images.tgz: 85%|████████▍ | 293M/345M [00:11<00:01, 30.0MB/s]
Downloading images.tgz: 86%|████████▌ | 297M/345M [00:11<00:01, 30.0MB/s]
Downloading images.tgz: 87%|████████▋ | 301M/345M [00:11<00:01, 29.9MB/s]
Downloading images.tgz: 88%|████████▊ | 305M/345M [00:11<00:01, 29.9MB/s]
Downloading images.tgz: 90%|████████▉ | 309M/345M [00:11<00:01, 29.9MB/s]
Downloading images.tgz: 91%|█████████ | 313M/345M [00:11<00:01, 29.9MB/s]
Downloading images.tgz: 92%|█████████▏| 317M/345M [00:12<00:00, 30.0MB/s]
Downloading images.tgz: 93%|█████████▎| 321M/345M [00:12<00:00, 30.0MB/s]
Downloading images.tgz: 94%|█████████▍| 325M/345M [00:12<00:00, 30.0MB/s]
Downloading images.tgz: 95%|█████████▌| 329M/345M [00:12<00:00, 30.0MB/s]
Downloading images.tgz: 97%|█████████▋| 333M/345M [00:12<00:00, 30.0MB/s]
Downloading images.tgz: 98%|█████████▊| 337M/345M [00:12<00:00, 29.9MB/s]
Downloading images.tgz: 99%|█████████▉| 341M/345M [00:12<00:00, 29.9MB/s]
Downloading images.tgz: 100%|██████████| 345M/345M [00:12<00:00, 26.7MB/s]
Extracting images.tgz: 0%| | 0/8190 [00:00<?, ?file/s]
Extracting images.tgz: 6%|▌ | 451/8190 [00:00<00:01, 4501.42file/s]
Extracting images.tgz: 11%|█ | 902/8190 [00:00<00:01, 4421.15file/s]
Extracting images.tgz: 16%|█▋ | 1348/8190 [00:00<00:01, 4436.17file/s]
Extracting images.tgz: 22%|██▏ | 1792/8190 [00:00<00:01, 4421.76file/s]
Extracting images.tgz: 27%|██▋ | 2235/8190 [00:00<00:01, 4421.88file/s]
Extracting images.tgz: 33%|███▎ | 2685/8190 [00:00<00:01, 4446.15file/s]
Extracting images.tgz: 38%|███▊ | 3132/8190 [00:00<00:01, 4453.87file/s]
Extracting images.tgz: 44%|████▎ | 3580/8190 [00:00<00:01, 4459.72file/s]
Extracting images.tgz: 49%|████▉ | 4026/8190 [00:00<00:00, 4453.86file/s]
Extracting images.tgz: 55%|█████▍ | 4476/8190 [00:01<00:00, 4466.53file/s]
Extracting images.tgz: 60%|██████ | 4923/8190 [00:01<00:00, 4462.99file/s]
Extracting images.tgz: 66%|██████▌ | 5370/8190 [00:01<00:00, 4463.07file/s]
Extracting images.tgz: 71%|███████ | 5817/8190 [00:01<00:00, 4426.02file/s]
Extracting images.tgz: 76%|███████▋ | 6260/8190 [00:01<00:00, 4418.49file/s]
Extracting images.tgz: 82%|████████▏ | 6704/8190 [00:01<00:00, 4422.21file/s]
Extracting images.tgz: 87%|████████▋ | 7147/8190 [00:01<00:00, 4398.37file/s]
Extracting images.tgz: 93%|█████████▎| 7593/8190 [00:01<00:00, 4415.77file/s]
Extracting images.tgz: 98%|█████████▊| 8036/8190 [00:01<00:00, 4418.18file/s]
Extracting images.tgz: 100%|██████████| 8190/8190 [00:01<00:00, 4431.29file/s]
Number of images in the gallery: 1538
Warning: You are sending unauthenticated requests to the HF Hub. Please set a HF_TOKEN to enable higher rate limits and faster downloads.
PosixPath('flower_image_store.safetensors')
Load the store back from disk¶
The saved file carries the embedder as well, so the store is rebuilt without touching the original images and without downloading the CLIP weights again. The index is rebuilt from the saved embeddings, using the parameters it was saved with.
This is the entry point for a fresh session. Run the import cell, then this one, and skip the embedding cell above entirely. A store loaded from disk is already built, so it needs no build_store() call.
One caveat applies to external indexes only. A FAISS index cannot be written into the file, so a rebuilt one has to be handed back as search_index=.... Without it the store falls back to an exact brute-force scan over the saved embeddings and warns about the fallback.
image_store = InMemoryImageEmbeddingStore.load_from_disk(
"flower_image_store.safetensors"
)
Helper to display images>¶
def _gallery_labels_by_path():
dataset = OxfordFlowerDataset()
return dict(zip(dataset.image_paths, dataset.labels, strict=True))
#: Class id of every gallery image, keyed by its path.
GALLERY_LABELS_BY_PATH = _gallery_labels_by_path()
def _as_rgb_array(image_array_or_path):
if isinstance(image_array_or_path, np.ndarray):
return image_array_or_path
with Image.open(image_array_or_path) as image:
return np.asarray(image.convert("RGB"))
def visualize_image(image_array_or_path, title=None):
plt.figure(figsize=(4, 4))
plt.imshow(_as_rgb_array(image_array_or_path))
plt.axis("off")
if title is not None:
plt.title(title)
plt.show()
def visualize_candidate(candidate):
label = GALLERY_LABELS_BY_PATH[candidate.path]
visualize_image(
candidate.path, title=f"label {label}, score = {candidate.score:.4f}"
)
The query image¶
The query is drawn from the test split, so it was never embedded into the gallery and the search has to generalise to an unseen image. Its own class label is shown above it, and it is the label every retrieved image is measured against further down.
test_dataset = OxfordFlowerDataset(purpose="test")
image1, label1, _ = test_dataset[172]
visualize_image(image1, title=f"query, label {label1}")
Retrieve the nearest neighbors¶
retrieve_top_k_similar embeds the query with the store’s own embedder, walks the index, and maps the returned row ids back to gallery paths.
It takes a batch as readily as a single image and always returns one ranked list per query, which is why the first list is unpacked with [0]. Batching is worth using whenever there is more than one query, because the index answers a single (M, D) matrix far faster than it answers M separate queries.
Each result is a Candidate, a named tuple of path and score, ordered best first. In cosine space the score is 1 - cosine_similarity, so lower means more similar and a near-duplicate of the query scores close to 0.0.
Query time is bounded by search_candidates, the width of the graph walk, which defaults to 50. Asking for more than search_candidates neighbors raises it to k automatically.
candidates = image_store.retrieve_top_k_similar(image1, k=5)[0]
Inspect the results¶
The top matches should carry the query’s own class label. HNSW is approximate, so a graph built for speed can miss a true neighbor. If the results look poor, rebuild with a larger graph_degree or build_candidates, or raise search_candidates at query time. Building the store with search_index=None gives the exact ranking to compare against, at the cost of scanning the whole gallery on every query.
for candidate in candidates:
visualize_candidate(candidate)
Refine the query with alpha query expansion¶
A query is a single point, and a single point can sit off to the side of the group it belongs to. Alpha query expansion (aQE) [7] moves it back towards the middle of that group before the ranking is decided.
The store searches once, reads the embeddings of the expansion_neighbors best matches back off the index, and replaces the query by the L2-normalised weighted average of itself and those matches. Each match is weighted by its cosine similarity to the query raised to expansion_alpha, so a close match pulls harder than a distant one, and a match whose similarity is not positive does not pull at all. The final search then runs with the refined query.
Arguments:
query_expansion: turns the refinement on. It is off by default, because it costs one extra index search per query plus the decoding ofexpansion_neighborsgallery vectors.expansion_alpha: exponent of the similarity weights.0weights every match with a positive similarity alike, which is the classic average query expansion.expansion_neighbors: how many of the top-ranked gallery images are averaged into the query.
The scores below are still the store’s own distances, so lower still means more similar, but they are measured from the refined query rather than from the one that was embedded and are not comparable to the scores above.
expanded_candidates = image_store.retrieve_top_k_similar(
image1,
k=5,
query_expansion=True,
expansion_alpha=3.0,
expansion_neighbors=50,
)[0]
for expanded_candidate in expanded_candidates:
visualize_candidate(expanded_candidate)
Re-rank the candidates with k-reciprocal encoding¶
Query expansion changes the query. Re-ranking leaves the query untouched and re-orders a pool of candidates that has already been retrieved.
KReciprocalReranker [8] asks of every candidate whether the query lies among its own nearest neighbors, not only whether it lies among the query’s. That relation is far stricter than plain proximity: a false match can sit close to the query while the query sits nowhere near the false match’s own neighbors. The query and every candidate are encoded into a k-reciprocal feature, a vector that holds a weight for each of their k-reciprocal neighbors and zero elsewhere, and the Jaccard distance between two such features says how much the two neighborhoods agree.
The neighborhoods are built among the candidates themselves, so a pool no larger than the answer leaves them nothing to say. A pool of 100 is retrieved below and the best five are kept.
Arguments:
k1: size of the neighborhoods the k-reciprocal sets are built from. The pool should hold at least this many candidates.k2: size of the neighborhood the local query expansion averages the features over.1turns that step off.lambda_value: weight of the original distance in the final one, from0(Jaccard distance only) to1(original ranking kept).top_k: how many of the re-ranked candidatesrerankreturns.
The scores are the final distances of the re-ranking rather than the store’s. They lie in [0, 1], and lower again means more similar.
Further reading: Re-ranking.
from pyvisim.retrieval.reranking import KReciprocalReranker
reranker = KReciprocalReranker(image_store, k1=20, k2=6, lambda_value=0.3)
candidate_pool = image_store.retrieve_top_k_similar(image1, k=100)[0]
reranked_candidates = reranker.rerank(candidate_pool, top_k=5)
for reranked_candidate in reranked_candidates:
visualize_candidate(reranked_candidate)
Search through a FAISS index¶
ExternalSearchIndex lets the store search through an index built by another library, so an algorithm this package does not ship can still be used over the same gallery. The example below rebuilds the HNSW graph with FAISS [5] and hands it over.
The graph is built over image_store.embeddings, the matrix the store’s own index holds, so the gallery is never embedded a second time.
Two things are the caller’s job here rather than the store’s:
The metric.
METRIC_INNER_PRODUCTranks by cosine similarity only if the vectors are L2-normalised. They already are, because the store was built in the default"cosine"space, and the queries arrive normalised too, becauseClipEmbeddernormalises what it returns.The parameters.
index_paramsis not forwarded to an external index.mis a constructor argument ofIndexHNSWFlat, whileefConstructionandefSearchare set on the graph itself.
An external index already holds the gallery, so a store on one is built as soon as it is constructed and needs no build_store() call.
from_faiss_index reads the gallery vectors back off the index, which IndexHNSWFlat can do because it keeps them uncompressed. A quantized index cannot, and needs them passed explicitly as the second argument.
import faiss
from pyvisim.retrieval.image_store import ExternalSearchIndex
gallery_vectors = image_store.embeddings
faiss_index = faiss.IndexHNSWFlat(
gallery_vectors.shape[1], 16, faiss.METRIC_INNER_PRODUCT
)
faiss_index.hnsw.efConstruction = 200
faiss_index.add(gallery_vectors)
faiss_index.hnsw.efSearch = 50
faiss_store = InMemoryImageEmbeddingStore(
image_paths=image_store.paths,
embedder=embedder,
search_index=ExternalSearchIndex.from_faiss_index(faiss_index, name="faiss-hnsw"),
)
faiss_store
InMemoryImageEmbeddingStore(num_images=1538, dim=512, index_name='faiss-hnsw', space='cosine')
Compare the two rankings¶
The two indexes report their scores in opposite units. FAISS was asked for inner products, so its score is a cosine similarity and higher means more similar. The built-in index returns the 1 - cosine_similarity distance of its "cosine" space, where lower means more similar. On L2-normalised vectors these are the same quantity read from opposite ends, so 1 - score converts either one into the other exactly.
A conversion is therefore applied below. The FAISS similarity is turned into a cosine distance with 1 - faiss_candidate.score before it is printed, so both columns speak the same units and can be read against each other directly. Neither store rescales anything of its own accord. The conversion is done here in the cell, and what the index itself returned is still faiss_candidate.score.
Both graphs are approximate and are built by different code, so the two rankings are free to disagree on a borderline neighbor. On this query they do not.
faiss_candidates = faiss_store.retrieve_top_k_similar(image1, k=5)[0]
for faiss_candidate, hnsw_candidate in zip(faiss_candidates, candidates, strict=True):
agreement = "same" if faiss_candidate.path == hnsw_candidate.path else "differs"
faiss_distance = 1.0 - faiss_candidate.score
print(
f"{agreement:>7} faiss={faiss_distance:.4f} "
f"hnsw={hnsw_candidate.score:.4f} {faiss_candidate.path}"
)
same faiss=0.0498 hnsw=0.0498 /home/runner/.cache/pyvisim/oxford_flower_dataset/images/jpg/image_00345.jpg
same faiss=0.0577 hnsw=0.0577 /home/runner/.cache/pyvisim/oxford_flower_dataset/images/jpg/image_00371.jpg
same faiss=0.0637 hnsw=0.0637 /home/runner/.cache/pyvisim/oxford_flower_dataset/images/jpg/image_00435.jpg
same faiss=0.0638 hnsw=0.0638 /home/runner/.cache/pyvisim/oxford_flower_dataset/images/jpg/image_00299.jpg
same faiss=0.0660 hnsw=0.0660 /home/runner/.cache/pyvisim/oxford_flower_dataset/images/jpg/image_00316.jpg
Serialize a store built on an external index¶
save_to_disk writes the embeddings, the paths and the embedder, but not the FAISS index itself, which this library cannot serialize. Loading therefore takes a rebuilt index back as search_index=..., and its name is checked against the saved one, so a mismatch is reported. Left out, the store falls back to an exact brute-force scan over the saved embeddings and warns about it.
A quantized index needs embeddings=... passed to save_to_disk as well, because what it reconstructs is an approximation of the embeddings rather than the embeddings themselves.
Note that the file written below is a second full copy of the gallery, roughly the size of the one written further up.
faiss_store.save_to_disk("flower_faiss_store.safetensors")
restored_store = InMemoryImageEmbeddingStore.load_from_disk(
"flower_faiss_store.safetensors",
search_index=ExternalSearchIndex.from_faiss_index(faiss_index, name="faiss-hnsw"),
)
restored_candidates = restored_store.retrieve_top_k_similar(image1, k=5)[0]
for restored_candidate in restored_candidates:
print(f"{restored_candidate.score:.4f} {restored_candidate.path}")
0.9502 /home/runner/.cache/pyvisim/oxford_flower_dataset/images/jpg/image_00345.jpg
0.9423 /home/runner/.cache/pyvisim/oxford_flower_dataset/images/jpg/image_00371.jpg
0.9363 /home/runner/.cache/pyvisim/oxford_flower_dataset/images/jpg/image_00435.jpg
0.9362 /home/runner/.cache/pyvisim/oxford_flower_dataset/images/jpg/image_00299.jpg
0.9340 /home/runner/.cache/pyvisim/oxford_flower_dataset/images/jpg/image_00316.jpg
Visualise the results¶
It can be observed that the results are identical to the ones returned by the built-in index, since the same hnsw parameters are used.
The scores in the titles are the ones FAISS itself returned rather than the distances converted further up, so here higher means more similar. The labels are read the same way as everywhere else.
Now, you can plug in any FAISS index you like for the search.
for faiss_candidate in faiss_candidates:
visualize_candidate(faiss_candidate)
References¶
[1] Nilsback, M.-E., & Zisserman, A. (2008). Automated Flower Classification over a Large Number of Classes. In Proceedings of the Sixth Indian Conference on Computer Vision, Graphics and Image Processing (ICVGIP), 722-729. https://www.robots.ox.ac.uk/~vgg/data/flowers/102/
[2] Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., Krueger, G., & Sutskever, I. (2021). Learning Transferable Visual Models From Natural Language Supervision. In Proceedings of the 38th International Conference on Machine Learning (ICML), PMLR 139, 8748-8763. https://arxiv.org/abs/2103.00020
[3] Malkov, Y. A., & Yashunin, D. A. (2020). Efficient and Robust Approximate Nearest Neighbor Search Using Hierarchical Navigable Small World Graphs. IEEE Transactions on Pattern Analysis and Machine Intelligence, 42(4), 824-836. https://arxiv.org/abs/1603.09320
[4] hnswlib, the reference implementation the compiled index is built on. https://github.com/nmslib/hnswlib
[5] Johnson, J., Douze, M., & Jegou, H. (2019). Billion-Scale Similarity Search with GPUs. IEEE Transactions on Big Data, 7(3), 535-547. https://arxiv.org/abs/1702.08734
[6] Pinecone. Hierarchical Navigable Small Worlds (HNSW). https://www.pinecone.io/learn/series/faiss/hnsw/
[7] Radenovic, F., Tolias, G., & Chum, O. (2019). Fine-tuning CNN Image Retrieval with No Human Annotation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 41(7), 1655-1668. https://arxiv.org/abs/1711.02512
[8] Zhong, Z., Zheng, L., Cao, D., & Li, S. (2017). Re-ranking Person Re-identification with k-reciprocal Encoding. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 1318-1327. https://arxiv.org/abs/1701.08398