DeepConvFeature

DeepConvFeature flattens the feature maps of one convolutional layer of a backbone into local descriptors. The VLAD and Fisher Vector embedders aggregate them the same way they aggregate SIFT descriptors.

API reference

class pyvisim.neural_networks.features.DeepConvFeature(backbone=None, target_submodule=None, layer_index=-1, device=None, transform=None, **kwargs)[source]

Bases: FeatureExtractorBase

Extracts convolutional feature maps from a chosen conv layer of a torchvision model. It flattens the feature maps into feature descriptors.

The concepts here were inspired by by the work on VLAD-DCNN features for face verification, as presented in [1], where VLAD embeddings were computed from deep convolutional features and input into a metric learning algorithm in order to distinguish between different people.

Parameters:

  • backbone (str | torch.nn.Module | None) –

    The convolutional backbone to extract features from. It may be:

    • None: builds a torchvision VGG16 with ImageNet weights.

    • A string naming a built-in backbone, e.g. "vgg16", which builds the torchvision model with its ImageNet weights. Every supported backbone is documented in https://mechacritter.github.io/Python-Visual-Similarity/neural_networks/backbones/backbones.html. To list them at runtime, use:

      from pyvisim.neural_networks.backbones import list_backbones

list_backbones()

    • A torch.nn.Module instance: any user-supplied PyTorch model.

    In the paper [1], a VGG-Face model trained on the Imdb-Wiki dataset was used with VLAD embedding for younger faces verification.

  • target_submodule (str | None) – Optional submodule name to hook into. If None, the whole model is used.

  • layer_index (int) – Which conv layer to hook (int). Use list_conv_layers(…) to see the ordering or use -1 for the last conv layer.

  • device (str | None) – ‘cpu’ or ‘cuda’. Where to run the model. None auto-selects ‘cuda’ when available, else ‘cpu’.

  • transform (transforms.Compose | None) – Optional torchvision.transforms.Compose. If None, images are converted to tensors and resized to 224x224.

  • kwargs (Any)

Deprecated since version 0.4.1: The model keyword argument is deprecated; pass the model through backbone instead. When model is supplied it is used as the backbone (unless backbone is also given) and a FutureWarning is emitted.

TODO

  • input range handling and batch processing; currently one image is processed per call.

References:

[1] Liangliang Wang and Deepu Rajan, “An Image Similarity Descriptor for Classification Tasks,” J. Vis. Commun. Image R., vol. 71, pp. 102847, 2020. [2] Weixia Zhang, Jia Yan, Wenxuan Shi, Tianpeng Feng, and Dexiang Deng, “Refining Deep Convolutional Features for Improving Fine-Grained Image Recognition,” EURASIP Journal on Image and Video Processing, 2017.

__call__(image, /, *, dims='HWC', value_range=(0.0, 255.0))[source]

Processes a single image through the chosen conv layer and returns flattened feature descriptors.

The input is normalized to a canonical uint8 (H, W, C) image and then passed through self.transform (which converts it to a tensor in [0, 1]).

For the batched version, use extract_batch() instead.

Parameters:

  • image (_Buffer | _SupportsArray[dtype[Any]] | _NestedSequence[_SupportsArray[dtype[Any]]] | bool | int | float | complex | str | bytes | _NestedSequence[bool | int | float | complex | str | bytes]) – Input image as MatLike (e.g. a NumPy (H, W, C) array or a torch (C, H, W) tensor; pass dims accordingly).

  • dims (str) – Axis-label string, one character per array axis in order: "H" = height (rows), "W" = width (columns), "C" = channels. For example, "HWC" is height × width × channels (NumPy/OpenCV layout); "CHW" is channels × height × width (PyTorch layout). See pyvisim.typing.

  • value_range (tuple[float, float]) – The (low, high) range the input values live in.

Returns:

N x D NumPy array, where N = (H_conv x W_conv) and D = number_of_channels.

Return type:

ndarray[tuple[int, …], dtype[float32]]

extract_batch(images, /, *, dims='HWC', value_range=(0.0, 255.0))[source]

Extracts the descriptors of a whole batch in a single forward pass.

The transform built for transform=None resizes every image to a fixed size, so the batch stacks into one tensor. A custom transform that preserves the input size does not, and the images are then extracted one at a time.

Parameters:
Returns:

One (Hf * Wf, D) descriptor array per input image.

Return type:

list[ndarray[tuple[int, …], dtype[float32]]]

classmethod from_dict(state, **kwargs)

Rebuilds an object from a state dictionary (see to_dict() to see the expected format).

Parameters:

  • state (dict[str, Any]) – A JSON-safe description of the object.

  • kwargs (Any) – Objects the state cannot describe, forwarded by load_from_disk(). Implementations that accept none raise an error if kwargs is not empty.

Returns:

A ready-to-use instance.

Return type:

FeatureExtractorBase

list_conv_layers()[source]

Utility function to collect convolutional layers (and sub-modules) from the model / chosen submodule.

Returns:

List of (layer_index, layer_module) for each convolutional layer.

Return type:

list[tuple[int, str, Conv2d]]

classmethod load_from_disk(path, **kwargs)

Loads an object previously saved with save_to_disk().

Not every part of an object survives serialization: an arbitrary callable such as a torchvision transform has no portable description, so it is left out of the file. Pass such an object back here as a keyword argument.

Parameters:

  • path (str | Path) – Path to the file to load.

  • kwargs (Any) – Objects the file cannot hold, forwarded to from_dict().

Returns:

A ready-to-use instance.

Raises:

  • FileNotFoundError – If path does not exist.

  • ValueError – If the file is not a valid file of this kind or was saved by a different class.

  • TypeError – If the class does not take one of kwargs.

Return type:

_SerializableT

save_to_disk(path)

Saves the serialized state of this object to a file.

Parameters:

path (str | Path) – Target file path. Overwritten if it exists.

Returns:

The path of the written file.

Raises:

OSError – If the destination directory does not exist.

Return type:

Path

to_dict()

Serializes this object into a JSON-safe state dictionary.

The mapping holds the output of _state() plus the format version under "format_version" and the class name under "__class__". Arrays may be embedded as __ndarray__ nodes, which the serialization layer stores as binary tensors.

Returns:

A JSON-safe description suitable for from_dict().

Return type:

dict[str, Any]

property output_dim: int

The dimensionality (D) of each feature vector, i.e., shape[1] of the output.