VLADEmbedder¶

VLAD (Vector of Locally Aggregated Descriptors) was introduced by Jegou et al. in 2010 as a compact alternative to bag-of-words for large-scale image retrieval. Similar to BoW, VLAD also uses K-Means for the cluster assignment of local features, but it aggregates the feature descriptors extracted from an image into a compact representation. Hence, the values of the descriptors themselves are considered, not just their occurrences.

VLAD computation¶

Given an image, the steps to encode it into a VLAD vector involve the following steps:

Feature Detection and Description: Extract descriptors from the image using a feature detector algorithm, like SIFT, RootSIFT, or SURF.
Dimension Reduction: Optionally, reduce the dimensionality of the descriptors using PCA to reduce memory usage and computation time.
Descriptor Assignment and Aggregation: Assign each descriptor to the nearest cluster center from a predefined set of centers obtained from a K-Means clustering model trained on the descriptors of the training set. Then, for each cluster center, aggregate the differences between the descriptors assigned to that cluster. Let \(x_i\) be a descriptor and \(c_k\) be the nearest cluster center. The assignment of \(x_i\) to \(c_k\) is computed as follows:

\[V_k = \sum_{x_i \in k} (x_i - c_k)\]

Where \(V_k\) is the aggregated vector for the cluster \(c_k\).
Concatenation: Concatenate all vectors \(V_k\) across all clusters to form the final VLAD vector.
L2 Normalization: Since two VLAD vectors are typically compared using distance metrics such as Euclidean distance or cosine similarity, it is essential to normalize both of them to have the same scale.

Usage¶

from pyvisim.classic import VLADEmbedder

vlad = VLADEmbedder(
    n_clusters=256,                  # number of visual words
    kmeans_params={"rng": 0},        # forwarded to  KMeans
    pca_params={"n_components": 64}, # optional, omit for no PCA
)
vlad.learn(images)                   # fits the PCA (if any) then K-Means

embedding = vlad.embed(image)        # Embed image into a VLAD vector

# Cosine similarity between two images
similarity = vlad.similarity_score(image1, image2)

vlad.save_to_disk("vlad.safetensors")   # Save the embedder to disk

# Load the embedder from disk
vlad = VLADEmbedder.load_from_disk("vlad.safetensors")

The resulting vector has shape (K * D,), where K is the number of clusters and D is the local descriptor dimension (after optional PCA).

References¶

R. Arandjelović and A. Zisserman. “All About VLAD”. In: 2013 IEEE Conference on Computer Vision and Pattern Recognition. 2013, pp. 1578-1585. doi: 10.1109/CVPR.2013.207.
H. Jégou, M. Douze, C. Schmid and P. Pérez, “Aggregating local descriptors into a compact image representation,” 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, San Francisco, CA, USA, 2010, pp. 3304-3311, doi: 10.1109/CVPR.2010.5540039.

API reference¶

class pyvisim.classic.VLADEmbedder(feature_extractor=None, n_clusters=256, kmeans_params=None, pca_params=None, power_norm_weight=1, norm_order=2, epsilon=1e-09, similarity_func='cosine', *, normalize=True, batch_size=16)[source]¶

Bases: ClusteringBasedEmbedder

This class embeds images into VLAD descriptor vectors using a chosen feature extractor and a K-Means clustering model, then compares two VLAD descriptor vectors with the configured similarity function.

The K-Means model is configured from the parameters passed to this constructor (n_clusters plus the optional kmeans_params dictionary) and fitted by calling learn(). An optional PCA model for dimensionality reduction is configured the same way via pca_params.

The output when calling embed has shape (num_clusters * feature_dim,).

You can use euclidean distance, manhattan distance, etc. as the similarity function.

The embedding can be used for indexing, retrieval, clustering or classification tasks.

For more information, see the documentation: https://mechacritter.github.io/Python-Visual-Similarity/classic/vlad/vlad.html.

Parameters:

feature_extractor (FeatureExtractorBase | None) – Feature extractor instance (should implement __call__). If None, RootSIFT is used.
n_clusters (int) – Number of K-Means clusters (visual words) to use.

kmeans_params (dict[str, Any] | None) –

Arguments for K-Means during vocabulary learning:

Parameter	Default	Meaning
`n_init`	`1`	Number of k-means++ seedings to run. The refined codebook with the lowest distortion is kept. Raise it for better, more stable vocabularies.
`thresh`	`1e-05`	Stops each refinement once the change in distortion drops below this (there is no maximum-iteration count).
`check_finite`	`True`	Whether to validate that the input contains only finite numbers. Turn it off for a small speed-up.
`rng`	`None`	Seed (`int`) or `numpy.random.Generator` for reproducible fitting.

pca_params (dict[str, Any] | None) –

Arguments for the Principal Component Analysis during vocabulary learning:

Parameter	Default	Meaning
`n_components`	(required)	Number of components to keep. Must be at most `min(n_samples, n_features)` of the training descriptors.
`whiten`	`False`	Scale each projected component to unit variance. Components with near-zero variance (rank-deficient descriptors) are floored at machine epsilon so the output stays finite.
`svd_solver`	`"auto"`	`"full"` (economy SVD), `"covariance_eigh"` (eigendecomposition of the feature covariance, fastest for many samples with few features), `"arpack"` (truncated SVD, computes only `n_components` singular triplets), or `"auto"`, which picks between them based on the training shape.
`tol`	`0.0`	Convergence tolerance of the `"arpack"` solver (0 means machine precision). Ignored by the other solvers.
`rng`	`None`	Seed (`int`) or `numpy.random.Generator` for the `"arpack"` solver’s starting vector. Ignored by the other solvers.

power_norm_weight (float) – Exponent for power normalization
norm_order (int) – Norm order for normalization.
epsilon (float) – Small constant to avoid division by zero.
similarity_func (str) – Name of the built-in similarity metric to use. One of "cosine", "euclidean", "l1" or "manhattan".
normalize (bool) – Whether embed L2-normalizes the embeddings it returns.
batch_size (int) – Maximum number of images processed in a single batch. Set to -1 to process all images as a single batch.

References

[1] Relja Arandjelović and Andrew Zisserman, ‘All About VLAD’, Department of Engineering Science, University of Oxford.
[2] Relja Arandjelović and Andrew Zisserman, “Three things everyone should know to improve object retrieval,” Department of Engineering Science, University of Oxford.
[3] Hervé Jégou, Florent Perronnin, Matthijs Douze, Jorge Sánchez, Patrick Pérez, and Cordelia Schmid, “Aggregating Local Image Descriptors into Compact Codes,” IEEE.

embed(images, *, dims='HWC', value_range=(0.0, 255.0))¶

Embeds one or more images into a batch of vector representations.

Each image is normalized to a canonical uint8 (H, W, C) array before feature extraction, so NumPy arrays, torch tensors and other array-like inputs are all accepted. When a batch axis is present (via dims), every image in the batch is embedded. The resulting vectors are L2-normalized row by row when normalize is True.

Parameters:

images (_Buffer | _SupportsArray[dtype[Any]] | _NestedSequence[_SupportsArray[dtype[Any]]] | bool | int | float | complex | str | bytes | _NestedSequence[bool | int | float | complex | str | bytes] | Iterable[_Buffer | _SupportsArray[dtype[Any]] | _NestedSequence[_SupportsArray[dtype[Any]]] | bool | int | float | complex | str | bytes | _NestedSequence[bool | int | float | complex | str | bytes]]) – A single MatLike image, a batched array, or an iterable of images. Consider using an iterator for large datasets.
dims (str) – Axis-label string, one character per array axis in order: "H" = height (rows), "W" = width (columns), "C" = channels (e.g. RGB), "B" = batch size. For example, "HWC" is height × width × channels (NumPy/OpenCV single-image layout); "CHW" is channels × height × width (PyTorch single-image layout); "BCHW" is batch × channels × height × width (PyTorch batched layout). See pyvisim.typing.
value_range (tuple[float, float]) – The (low, high) range the input values live in; converted into the canonical [0, 255] range.

Returns:

vector representations of the given images, L2-normalized row by row if normalize is True.

Raises:

ValueError – If images holds no image.

Return type:

ndarray[tuple[int, …], dtype[floating[Any]]]

classmethod from_dict(state, *, feature_extractor_params=None, **kwargs)¶

Rebuilds the embedder a state dictionary describes.

Parameters:

state (dict[str, Any]) – A JSON-safe embedder description.
feature_extractor_params (dict[str, Any] | None) – Objects the state of the feature extractor cannot describe, forwarded to from_dict().
kwargs (Any) – Not supported, must be empty.

Returns:

The reconstructed embedder.

Raises:

TypeError – If kwargs is not empty, or the feature extractor does not take one of feature_extractor_params.
ValueError – If the feature extractor cannot be rebuilt from its state and feature_extractor_params.

Return type:

_ClusteringEmbedderT

learn(images, /, *, dim_reduction_factor=None, dims='HWC', value_range=(0.0, 255.0))¶

Learns the visual vocabulary from the given images.

The clustering model configured at initialization is fitted on the extracted features. If a PCA model is configured, the features are reduced with it first (fitting it beforehand if necessary).

Parameters:

images (_Buffer | _SupportsArray[dtype[Any]] | _NestedSequence[_SupportsArray[dtype[Any]]] | bool | int | float | complex | str | bytes | _NestedSequence[bool | int | float | complex | str | bytes] | Iterable[_Buffer | _SupportsArray[dtype[Any]] | _NestedSequence[_SupportsArray[dtype[Any]]] | bool | int | float | complex | str | bytes | _NestedSequence[bool | int | float | complex | str | bytes]]) – A single MatLike image, a batched array, or an iterable of images. Each image is normalized to a canonical uint8 (H, W, C) array before feature extraction.
dim_reduction_factor (int | None) – If a value is provided, a new PCA model will be used to reduce the dimensionality of the feature space
dims (str) – Axis-label string, one character per array axis in order: "H" = height (rows), "W" = width (columns), "C" = channels (e.g. RGB), "B" = batch size. For example, "HWC" is height × width × channels (NumPy/OpenCV single-image layout); "CHW" is channels × height × width (PyTorch single-image layout); "BCHW" is batch × channels × height × width (PyTorch batched layout). See pyvisim.typing.
value_range (tuple[float, float]) – The (low, high) range the input values live in; converted into the canonical [0, 255] range.

Raises:

RuntimeError – If the embedder has no clustering model configured.
ValueError – If dim_reduction_factor is provided but is not a positive integer.

Return type:

None

load_clustering_model_from_sklearn(model)¶

Replaces this embedder’s clustering model with one created from a scikit-learn estimator.

The estimator’s constructor arguments are translated by the clustering model class’ from_sklearn method; if the estimator is fitted, its learned state is adopted and validated against the configured feature extractor / PCA. Only clustering model classes exposing from_sklearn support this — currently KMeans (VLAD) and DiagCovarGaussianMixture (Fisher Vector).

Parameters:

model (Any) – A scikit-learn estimator of the type matching this embedder’s clustering model (e.g. sklearn.cluster.KMeans for VLAD), fitted or not.

Raises:

NotImplementedError – If this embedder’s clustering model class cannot be created from a scikit-learn estimator.
TypeError – If model is not of the supported estimator type.
RuntimeError – If the fitted estimator’s input size is incompatible with the configured feature extractor or PCA.

Return type:

None

classmethod load_from_disk(path, **kwargs)¶

Loads an object previously saved with save_to_disk().

Not every part of an object survives serialization: an arbitrary callable such as a torchvision transform has no portable description, so it is left out of the file. Pass such an object back here as a keyword argument.

Parameters:

path (str | Path) – Path to the file to load.
kwargs (Any) – Objects the file cannot hold, forwarded to from_dict().

Returns:

A ready-to-use instance.

Raises:

FileNotFoundError – If path does not exist.
ValueError – If the file is not a valid file of this kind or was saved by a different class.
TypeError – If the class does not take one of kwargs.

Return type:

_SerializableT

save_to_disk(path)¶

Saves the serialized state of this object to a file.

Parameters:: path (str | Path) – Target file path. Overwritten if it exists.
Returns:: The path of the written file.
Raises:: OSError – If the destination directory does not exist.
Return type:: Path

set_batch_size(batch_size)¶

Sets the number of items processed per batch.

Parameters:: batch_size (int) – Maximum number of images processed in a single batch. Set to -1 to process all images as a single batch.
Raises:: ValueError – If batch_size is neither -1 nor a positive integer.
Return type:: None

similarity_score(images1, images2, *, dims='HWC', value_range=(0.0, 255.0))¶

Compute the similarity scores matrix between two (batches of) images.

Parameters:

images1 (_Buffer | _SupportsArray[dtype[Any]] | _NestedSequence[_SupportsArray[dtype[Any]]] | bool | int | float | complex | str | bytes | _NestedSequence[bool | int | float | complex | str | bytes] | Iterable[_Buffer | _SupportsArray[dtype[Any]] | _NestedSequence[_SupportsArray[dtype[Any]]] | bool | int | float | complex | str | bytes | _NestedSequence[bool | int | float | complex | str | bytes]]) – First (batch of) image(s) as MatLike (NumPy array, torch tensor or array-like).
images2 (_Buffer | _SupportsArray[dtype[Any]] | _NestedSequence[_SupportsArray[dtype[Any]]] | bool | int | float | complex | str | bytes | _NestedSequence[bool | int | float | complex | str | bytes] | Iterable[_Buffer | _SupportsArray[dtype[Any]] | _NestedSequence[_SupportsArray[dtype[Any]]] | bool | int | float | complex | str | bytes | _NestedSequence[bool | int | float | complex | str | bytes]]) – Second (batch of) image(s) as MatLike.
dims (str) – Axis-label string, one character per array axis in order: "H" = height (rows), "W" = width (columns), "C" = channels (e.g. RGB), "B" = batch size. For example, "HWC" is height × width × channels (NumPy/OpenCV single-image layout); "CHW" is channels × height × width (PyTorch single-image layout); "BCHW" is batch × channels × height × width (PyTorch batched layout). See pyvisim.typing.
value_range (tuple[float, float]) – The (low, high) range the input values live in; converted into the canonical [0, 255] range.

Returns:

The similarity score matrix of shape (len(images1), len(images2)).

Return type:

ndarray[tuple[int, …], dtype[float32]]

to_dict()¶

Serializes this object into a JSON-safe state dictionary.

The mapping holds the output of _state() plus the format version under "format_version" and the class name under "__class__". Arrays may be embedded as __ndarray__ nodes, which the serialization layer stores as binary tensors.

Returns:: A JSON-safe description suitable for from_dict().
Return type:: dict[str, Any]

property normalize: bool¶: Whether the embeddings returned by embed() are L2-normalized.

property similarity_func: Callable[[ndarray[tuple[int, ...], dtype[floating[Any]]], ndarray[tuple[int, ...], dtype[floating[Any]]]], ndarray[tuple[int, ...], dtype[floating[Any]]]]¶: The resolved similarity function callable.

property similarity_func_name: str¶: The name of the configured similarity metric (e.g. "cosine").