FisherVectorEmbedder

The Fisher Vector solves both problems of the BoW model. First, it encodes higher-order statistics, such as the first and optionally second-order differences, instead of just counting the occurrences of visual words like BoW. This method is derived from the Fisher kernel framework, which describes a sample set’s deviation from an average distribution. Secondly, the distribution of the local descriptors, unlike BoW and VLAD, is modeled by a Gaussian Mixture Model. This mitigates the hard assignment problem introduced by the K-Means algorithm, since each descriptor is assigned to multiple Gaussian components with different probabilities.

Computation

Fisher kernel framework

Given a set of \(T\) local descriptors \(X = \{x_t; t = 1, \ldots, T\}\) extracted from an image, it is assumed that the generation process of \(X\) can be modeled by an image-independent probability density function \(u_{\lambda}\) with parameters \(\lambda\) [Jégou et al., 2012]. The gradient vector \(G^{X}_{\lambda}\) is obtained by computing the gradient of the log-likelihood of the sample set \(X\) with respect to the parameters \(\lambda\):

\[G^{X}_{\lambda} = \frac{1}{T} \nabla_{\lambda} \log u_{\lambda}(X)\]

where \(G^{X}_{\lambda}\) describes the contribution of the parameters to the generation process [Perronnin & Dance, 2010].

The Fisher kernel is then defined as:

\[K(X, Y) = (G^{X}_{\lambda})^T F_{\lambda}^{-1} G^{Y}_{\lambda}\]

where \(F_{\lambda}\) is the Fisher information matrix, defined by:

\[F_{\lambda} = \mathbb{E}_{x \sim u_{\lambda}} \left[ \nabla_{\lambda} \log u_{\lambda}(x) \nabla_{\lambda} \log u_{\lambda}(x)^T \right]\]

\(\mathcal{G}^{X}_{\lambda}\) is the Fisher Vector after applying the Cholesky decomposition on \(F_{\lambda}^{-1} = L_{\lambda}^T L_{\lambda}\), and is computed as:

\[\mathcal{G}_i^X = L_{\lambda} G^{X}_{\lambda}\]

Fisher Vector computation

As discussed, the Fisher Vector encodes each descriptor to multiple Gaussian components (also called “soft assignment”). The probability of a descriptor \(x_t\) belonging to the \(i\)-th Gaussian is computed with the Gaussian Mixture Model.

The Gaussian Mixture Model is chosen for \(u_{\lambda}(x) = \sum_{i=1}^{K} w_i u_i(x)\), where \(w_i, \mu_i, \Sigma_i\) are the mixture weights, mean vectors, and variance matrices of the Gaussian \(u_i\). The Fisher Vector is then computed as:

\[\gamma_t(i) = \frac{w_i u_i(x_t)}{\sum_{j=1}^{K} w_j u_j(x_t)}\]
\[\mathcal{G}_i^X = \frac{1}{T \sqrt{w_i}} \sum_{t=1}^{T} \gamma_t(i)\, \sigma_i^{-1} (x_t - \mu_i)\]

where:

  • \(\gamma_t(i)\) is the soft assignment of descriptor \(x_t\) to the \(i\)-th Gaussian.

  • \(w_i\), \(\mu_i\), and \(\Sigma_i\) are the mixture weight, mean vector, and covariance matrix of the \(i\)-th Gaussian component.

The final Fisher Vector \(G^{X}_{\lambda}\) is the concatenation of the vectors \(G^{X}_{i}\) for \(i = 1, \ldots, K\), resulting in a \(K \times d\)-dimensional vector. This vector captures both the occurrence and distributional properties of the local descriptors.

The resulting vector has shape (2 * K * D + K,), where K is the number of GMM components and D is the local descriptor dimension (after optional PCA).

Usage

from pyvisim.classic import FisherVectorEmbedder

fisher = FisherVectorEmbedder(
    n_components=256,                # number of mixture components
    gmm_params={"rng": 0},           # forwarded to the GMM
    pca_params={"n_components": 64}, # optional, omit for no PCA
)
fisher.learn(images)                 # fits the PCA (if any) then the GMM

embedding = fisher.embed(image)      # Embed image into a Fisher Vector

# Cosine similarity between two images
similarity = fisher.similarity_score(image1, image2)

fisher.save_to_disk("fisher.safetensors")   # Save the embedder to disk

# Load the embedder from disk
fisher = FisherVectorEmbedder.load_from_disk("fisher.safetensors")

References

  • H. Jégou et al. “Aggregating Local Image Descriptors into Compact Codes”. In: IEEE Transactions on Pattern Analysis and Machine Intelligence 34.9 (2012), pp. 1704-1716. doi: 10.1109/TPAMI.2011.235.

API reference

class pyvisim.classic.FisherVectorEmbedder(feature_extractor=None, n_components=256, gmm_params=None, pca_params=None, power_norm_weight=0.5, norm_order=2, epsilon=1e-09, similarity_func='cosine', *, normalize=True, batch_size=16)[source]

Bases: ClusteringBasedEmbedder

This class serves as an embedder that transforms input images into Fisher Vector descriptors.

The Fisher Vector representation is based on the gradients of the GMM parameters (weights, means, and covariances) with respect to the feature descriptors extracted from the images. The representation is optionally power-normalized and L2-normalized.

The Gaussian Mixture Model is configured from the parameters passed to this constructor (n_components plus the optional gmm_params dictionary) and fitted by calling learn(). An optional PCA model for dimensionality reduction is configured the same way via pca_params.

The output when calling embed has shape (2 * num_clusters * feature_dim + num_clusters,).

For more information, see the documentation: https://mechacritter.github.io/Python-Visual-Similarity/classic/fisher_vector/fisher_vector.html.

Parameters:

  • feature_extractor (FeatureExtractorBase | None) – Feature extractor instance. If None, RootSIFT is used.

  • n_components (int) – Number of Gaussian mixture components (visual words) to use.

  • gmm_params (dict[str, Any] | None) –

    Arguments for Gaussian Mixture Model during vocabulary learning:

    Parameter

    Default

    Meaning

    n_init

    1

    Number of k-means++ seeded EM runs. The run with the highest final log-likelihood is kept. Raise it for better, more stable vocabularies.

    max_iter

    100

    Maximum number of EM iterations per run.

    tol

    1e-3

    Convergence threshold: a run stops when the change of the mean per-sample log-likelihood between iterations falls below it.

    reg_covar

    1e-6

    Non-negative regularisation added to (and floored on) the per-feature variances, keeping them strictly positive when a component collapses or dies.

    rng

    None

    Seed (int) or numpy.random.Generator for reproducible fitting.

  • pca_params (dict[str, Any] | None) –

    Arguments for the Principal Component Analysis during vocabulary learning:

    Parameter

    Default

    Meaning

    n_components

    (required)

    Number of components to keep. Must be at most min(n_samples, n_features) of the training descriptors.

    whiten

    False

    Scale each projected component to unit variance. Components with near-zero variance (rank-deficient descriptors) are floored at machine epsilon so the output stays finite.

    svd_solver

    "auto"

    "full" (economy SVD), "covariance_eigh" (eigendecomposition of the feature covariance, fastest for many samples with few features), "arpack" (truncated SVD, computes only n_components singular triplets), or "auto", which picks between them based on the training shape.

    tol

    0.0

    Convergence tolerance of the "arpack" solver (0 means machine precision). Ignored by the other solvers.

    rng

    None

    Seed (int) or numpy.random.Generator for the "arpack" solver’s starting vector. Ignored by the other solvers.

  • power_norm_weight (float) – Exponent for power normalization

  • norm_order (int) – Norm order for normalization.

  • epsilon (float) – Small constant to avoid division by zero.

  • similarity_func (str) – Name of the built-in similarity metric to use. One of "cosine", "euclidean", "l1" or "manhattan".

  • normalize (bool) – Whether embed L2-normalizes the embeddings it returns.

  • batch_size (int) – Maximum number of images processed in a single batch. Set to -1 to process all images as a single batch.

References

  • [1] Hervé Jégou, Florent Perronnin, Matthijs Douze, Jorge Sánchez, Patrick Pérez, and Cordelia Schmid, “Aggregating Local Image Descriptors into Compact Codes,” IEEE.

embed(images, *, dims='HWC', value_range=(0.0, 255.0))

Embeds one or more images into a batch of vector representations.

Each image is normalized to a canonical uint8 (H, W, C) array before feature extraction, so NumPy arrays, torch tensors and other array-like inputs are all accepted. When a batch axis is present (via dims), every image in the batch is embedded. The resulting vectors are L2-normalized row by row when normalize is True.

Parameters:

  • images (_Buffer | _SupportsArray[dtype[Any]] | _NestedSequence[_SupportsArray[dtype[Any]]] | bool | int | float | complex | str | bytes | _NestedSequence[bool | int | float | complex | str | bytes] | Iterable[_Buffer | _SupportsArray[dtype[Any]] | _NestedSequence[_SupportsArray[dtype[Any]]] | bool | int | float | complex | str | bytes | _NestedSequence[bool | int | float | complex | str | bytes]]) – A single MatLike image, a batched array, or an iterable of images. Consider using an iterator for large datasets.

  • dims (str) – Axis-label string, one character per array axis in order: "H" = height (rows), "W" = width (columns), "C" = channels (e.g. RGB), "B" = batch size. For example, "HWC" is height × width × channels (NumPy/OpenCV single-image layout); "CHW" is channels × height × width (PyTorch single-image layout); "BCHW" is batch × channels × height × width (PyTorch batched layout). See pyvisim.typing.

  • value_range (tuple[float, float]) – The (low, high) range the input values live in; converted into the canonical [0, 255] range.

Returns:

vector representations of the given images, L2-normalized row by row if normalize is True.

Raises:

ValueError – If images holds no image.

Return type:

ndarray[tuple[int, …], dtype[floating[Any]]]

classmethod from_dict(state, *, feature_extractor_params=None, **kwargs)

Rebuilds the embedder a state dictionary describes.

Parameters:

  • state (dict[str, Any]) – A JSON-safe embedder description.

  • feature_extractor_params (dict[str, Any] | None) – Objects the state of the feature extractor cannot describe, forwarded to from_dict().

  • kwargs (Any) – Not supported, must be empty.

Returns:

The reconstructed embedder.

Raises:

  • TypeError – If kwargs is not empty, or the feature extractor does not take one of feature_extractor_params.

  • ValueError – If the feature extractor cannot be rebuilt from its state and feature_extractor_params.

Return type:

_ClusteringEmbedderT

learn(images, /, *, dim_reduction_factor=None, dims='HWC', value_range=(0.0, 255.0))

Learns the visual vocabulary from the given images.

The clustering model configured at initialization is fitted on the extracted features. If a PCA model is configured, the features are reduced with it first (fitting it beforehand if necessary).

Parameters:

  • images (_Buffer | _SupportsArray[dtype[Any]] | _NestedSequence[_SupportsArray[dtype[Any]]] | bool | int | float | complex | str | bytes | _NestedSequence[bool | int | float | complex | str | bytes] | Iterable[_Buffer | _SupportsArray[dtype[Any]] | _NestedSequence[_SupportsArray[dtype[Any]]] | bool | int | float | complex | str | bytes | _NestedSequence[bool | int | float | complex | str | bytes]]) – A single MatLike image, a batched array, or an iterable of images. Each image is normalized to a canonical uint8 (H, W, C) array before feature extraction.

  • dim_reduction_factor (int | None) – If a value is provided, a new PCA model will be used to reduce the dimensionality of the feature space

  • dims (str) – Axis-label string, one character per array axis in order: "H" = height (rows), "W" = width (columns), "C" = channels (e.g. RGB), "B" = batch size. For example, "HWC" is height × width × channels (NumPy/OpenCV single-image layout); "CHW" is channels × height × width (PyTorch single-image layout); "BCHW" is batch × channels × height × width (PyTorch batched layout). See pyvisim.typing.

  • value_range (tuple[float, float]) – The (low, high) range the input values live in; converted into the canonical [0, 255] range.

Raises:

  • RuntimeError – If the embedder has no clustering model configured.

  • ValueError – If dim_reduction_factor is provided but is not a positive integer.

Return type:

None

load_clustering_model_from_sklearn(model)

Replaces this embedder’s clustering model with one created from a scikit-learn estimator.

The estimator’s constructor arguments are translated by the clustering model class’ from_sklearn method; if the estimator is fitted, its learned state is adopted and validated against the configured feature extractor / PCA. Only clustering model classes exposing from_sklearn support this — currently KMeans (VLAD) and DiagCovarGaussianMixture (Fisher Vector).

Parameters:

model (Any) – A scikit-learn estimator of the type matching this embedder’s clustering model (e.g. sklearn.cluster.KMeans for VLAD), fitted or not.

Raises:

  • NotImplementedError – If this embedder’s clustering model class cannot be created from a scikit-learn estimator.

  • TypeError – If model is not of the supported estimator type.

  • RuntimeError – If the fitted estimator’s input size is incompatible with the configured feature extractor or PCA.

Return type:

None

classmethod load_from_disk(path, **kwargs)

Loads an object previously saved with save_to_disk().

Not every part of an object survives serialization: an arbitrary callable such as a torchvision transform has no portable description, so it is left out of the file. Pass such an object back here as a keyword argument.

Parameters:

  • path (str | Path) – Path to the file to load.

  • kwargs (Any) – Objects the file cannot hold, forwarded to from_dict().

Returns:

A ready-to-use instance.

Raises:

  • FileNotFoundError – If path does not exist.

  • ValueError – If the file is not a valid file of this kind or was saved by a different class.

  • TypeError – If the class does not take one of kwargs.

Return type:

_SerializableT

save_to_disk(path)

Saves the serialized state of this object to a file.

Parameters:

path (str | Path) – Target file path. Overwritten if it exists.

Returns:

The path of the written file.

Raises:

OSError – If the destination directory does not exist.

Return type:

Path

set_batch_size(batch_size)

Sets the number of items processed per batch.

Parameters:

batch_size (int) – Maximum number of images processed in a single batch. Set to -1 to process all images as a single batch.

Raises:

ValueError – If batch_size is neither -1 nor a positive integer.

Return type:

None

similarity_score(images1, images2, *, dims='HWC', value_range=(0.0, 255.0))

Compute the similarity scores matrix between two (batches of) images.

Parameters:
Returns:

The similarity score matrix of shape (len(images1), len(images2)).

Return type:

ndarray[tuple[int, …], dtype[float32]]

to_dict()

Serializes this object into a JSON-safe state dictionary.

The mapping holds the output of _state() plus the format version under "format_version" and the class name under "__class__". Arrays may be embedded as __ndarray__ nodes, which the serialization layer stores as binary tensors.

Returns:

A JSON-safe description suitable for from_dict().

Return type:

dict[str, Any]

property normalize: bool

Whether the embeddings returned by embed() are L2-normalized.

property similarity_func: Callable[[ndarray[tuple[int, ...], dtype[floating[Any]]], ndarray[tuple[int, ...], dtype[floating[Any]]]], ndarray[tuple[int, ...], dtype[floating[Any]]]]

The resolved similarity function callable.

property similarity_func_name: str

The name of the configured similarity metric (e.g. "cosine").