TripletNeuralNetwork

A triplet network is a shared-weight embedding network trained on triplets of anchor, positive (same class) and negative (different class) images: the anchor is pulled towards the positive and pushed away from the negative by at least a margin. The three classic “branches” of the architecture are realized implicitly by weight sharing: every image is passed through the same backbone and projection head, and the embeddings are L2-normalized so that cosine similarity reduces to a dot product. Following diagram visualizes this:

Anchor   ───┐
            │    ┌──────────┐    ┌────────────────┐    ┌──────────────┐
Positive ───┼───►│ Backbone │───►│ Embedding Head │───►│ L2 Normalize │───► Embeddings
            │    └──────────┘    └────────────────┘    └──────────────┘
Negative ───┘    (Shared Weights)

              Embedding A + Embedding P + Embedding N
                                 │
                                 ▼
┌─────────────────────────────────────────────────────────────────┐
│ Triplet Loss (training) / fixed metric, e.g. cosine (inference) │
└─────────────────────────────────────────────────────────────────┘

Triplet loss is used to train this network, which has the formula:

\[L(a, p, n) = \max\bigl(0, \, d(a, p) - d(a, n) + m\bigr)\]

Training with online mining

See this tutorial.

Mining strategies

Below mining strategies are implemented in the TripletLoss:

mining

What it picks

Averaged over

Memory

"semi_hard" (default)

For every positive pair, the closest negative that is still farther than the positive. Falls back to the anchor’s farthest negative when there is none.

all positive pairs

O(B³)

"batch_hard"

Per anchor, its farthest positive and its closest negative (Hermans et al., 2017).

anchors with both

O(B²)

"batch_all"

Every valid triplet, but only the ones that violate the margin count towards the mean. Averaging over all of them would let the trivially satisfied majority wash out the signal.

violating triplets

O(B³)

Saving and loading

Save the model to disk:

path = model.save_to_disk("triplet_resnet18.safetensors")
model = TripletNeuralNetwork.load_from_disk(path)

Load the model from disk. Note that the transform has to be passed in again because they are not JSON-serializable:

model = TripletNeuralNetwork.load_from_disk(path, transform=my_transform)

References

  1. Deep Metric Learning Using Triplet Network (Hoffer & Ailon, 2014) https://arxiv.org/abs/1412.6622

  2. FaceNet: A Unified Embedding for Face Recognition and Clustering (Schroff, Kalenichenko, & Philbin, 2015) https://doi.org/10.1109/CVPR.2015.7298682

  3. In Defense of the Triplet Loss for Person Re-Identification (Hermans, Beyer, & Leibe, 2017) https://arxiv.org/abs/1703.07737

API reference

class pyvisim.neural_networks.TripletNeuralNetwork(backbone='resnet18', embedding_dim=128, similarity_func='cosine', transform=None, device='cpu', pretrained_backbone=True, *, normalize=True, batch_size=16)[source]

Bases: BackboneWithHead

Triplet network for image similarity, proposed in Hoffer, E., & Ailon, N. (2014). Deep Metric Learning Using Triplet Network and popularized by Schroff, F., Kalenichenko, D., & Philbin, J. (2015). FaceNet: A Unified Embedding for Face Recognition and Clustering.

For more information, see the documentation: https://mechacritter.github.io/Python-Visual-Similarity/neural_networks/triplet/triplet.html.

Training follows FaceNet’s online mining scheme exclusively: instead of preparing (anchor, positive, negative) files offline, a labeled batch of images is passed through forward() once and pyvisim.neural_networks.losses.TripletLoss mines the triplets from the batch itself.

References:

[1] Hoffer, E., & Ailon, N. (2014). Deep Metric Learning Using Triplet Network. https://arxiv.org/abs/1412.6622

[2] Schroff, F., Kalenichenko, D., & Philbin, J. (2015). FaceNet: A Unified Embedding for Face Recognition and Clustering. CVPR. https://doi.org/10.1109/CVPR.2015.7298682

param backbone:

name of feature-extraction network. See https://mechacritter.github.io/Python-Visual-Similarity/neural_networks/backbones/backbones.html.

param embedding_dim:

Dimensionality of the projected embedding space.

param similarity_func:

Name of the built-in similarity metric used to score two embeddings. One of "cosine", "euclidean", "l1" or "manhattan".

param transform:

processing transform applied to every input image. If None, the ImageNet preprocessing matching the backbone is used.

param device:

Device on which the model is placed.

param pretrained_backbone:

Whether to use a backbone pretrained on ImageNet. If you are loading the TripletNeuralNetwork from a checkpoint, set this to False to avoid downloading the weights again.

param normalize:

Whether embed L2-normalizes the embeddings it returns.

param batch_size:

Maximum number of images processed in a single batch. Set to -1 to process all images as a single batch.

raises ValueError:

If embedding_dim is not a positive integer, if backbone is not a supported backbone name, or if similarity_func is not a supported similarity metric.

embed(images, *, dims='HWC', value_range=(0.0, 255.0))

Embeds one or more images into a batch of vector representations.

Each image is normalized to a canonical uint8 (H, W, C) array before feature extraction, so NumPy arrays, torch tensors and other array-like inputs are all accepted. When a batch axis is present (via dims), every image in the batch is embedded. The resulting vectors are L2-normalized row by row when normalize is True.

Parameters:

  • images (_Buffer | _SupportsArray[dtype[Any]] | _NestedSequence[_SupportsArray[dtype[Any]]] | bool | int | float | complex | str | bytes | _NestedSequence[bool | int | float | complex | str | bytes] | Iterable[_Buffer | _SupportsArray[dtype[Any]] | _NestedSequence[_SupportsArray[dtype[Any]]] | bool | int | float | complex | str | bytes | _NestedSequence[bool | int | float | complex | str | bytes]]) – A single MatLike image, a batched array, or an iterable of images. Consider using an iterator for large datasets.

  • dims (str) – Axis-label string, one character per array axis in order: "H" = height (rows), "W" = width (columns), "C" = channels (e.g. RGB), "B" = batch size. For example, "HWC" is height × width × channels (NumPy/OpenCV single-image layout); "CHW" is channels × height × width (PyTorch single-image layout); "BCHW" is batch × channels × height × width (PyTorch batched layout). See pyvisim.typing.

  • value_range (tuple[float, float]) – The (low, high) range the input values live in; converted into the canonical [0, 255] range.

Returns:

vector representations of the given images, L2-normalized row by row if normalize is True.

Raises:

ValueError – If images holds no image.

Return type:

ndarray[tuple[int, …], dtype[floating[Any]]]

forward(x)[source]

Computes L2-normalized embeddings for a batch of preprocessed images.

During training this single shared-weight pass replaces the three explicit triplet branches: feed a labeled batch through it and mine the triplets online with pyvisim.neural_networks.losses.TripletLoss.

Parameters:

x (Tensor) – Preprocessed image tensor of shape (batch, channels, H, W).

Returns:

L2-normalized embeddings of shape (batch, embedding_dim).

Return type:

Tensor

classmethod from_dict(state, **kwargs)

Rebuilds the embedder a state dictionary describes.

Called on SerializableImageEmbedder itself, it hands the state to the from_dict of the class named under "__class__", so a state can be rebuilt without knowing which embedder wrote it.

Parameters:

  • state (dict[str, Any]) – A JSON-safe embedder description.

  • kwargs (Any) – Objects the state cannot describe, forwarded to the embedder’s own from_dict.

Returns:

The reconstructed embedder.

Raises:

  • ValueError – If state names no concrete embedder class.

  • NotImplementedError – If called on a subclass that does not implement its own from_dict.

Return type:

_NeuralEmbedderT

classmethod load_from_disk(path, **kwargs)

Loads an object previously saved with save_to_disk().

Not every part of an object survives serialization: an arbitrary callable such as a torchvision transform has no portable description, so it is left out of the file. Pass such an object back here as a keyword argument.

Parameters:

  • path (str | Path) – Path to the file to load.

  • kwargs (Any) – Objects the file cannot hold, forwarded to from_dict().

Returns:

A ready-to-use instance.

Raises:

  • FileNotFoundError – If path does not exist.

  • ValueError – If the file is not a valid file of this kind or was saved by a different class.

  • TypeError – If the class does not take one of kwargs.

Return type:

_SerializableT

save_to_disk(path)

Saves the serialized state of this object to a file.

Parameters:

path (str | Path) – Target file path. Overwritten if it exists.

Returns:

The path of the written file.

Raises:

OSError – If the destination directory does not exist.

Return type:

Path

set_batch_size(batch_size)

Sets the number of items processed per batch.

Parameters:

batch_size (int) – Maximum number of images processed in a single batch. Set to -1 to process all images as a single batch.

Raises:

ValueError – If batch_size is neither -1 nor a positive integer.

Return type:

None

similarity_score(images1, images2, *, dims='HWC', value_range=(0.0, 255.0))

Compute the similarity scores matrix between two (batches of) images.

Parameters:
Returns:

The similarity score matrix of shape (len(images1), len(images2)).

Return type:

ndarray[tuple[int, …], dtype[float32]]

to_dict()

Serializes this object into a JSON-safe state dictionary.

The mapping holds the output of _state() plus the format version under "format_version" and the class name under "__class__". Arrays may be embedded as __ndarray__ nodes, which the serialization layer stores as binary tensors.

Returns:

A JSON-safe description suitable for from_dict().

Return type:

dict[str, Any]

property backbone: Module

The shared feature-extraction backbone.

property device: device

The device the model’s parameters live on.

Derived from the parameters themselves rather than cached, so it stays correct after the user moves the model with model.to(...).

property head: Module

The projection head mapping backbone features to embeddings.

property normalize: bool

Whether the embeddings returned by embed() are L2-normalized.

property similarity_func: Callable[[ndarray[tuple[int, ...], dtype[floating[Any]]], ndarray[tuple[int, ...], dtype[floating[Any]]]], ndarray[tuple[int, ...], dtype[floating[Any]]]]

The resolved similarity function callable.

property similarity_func_name: str

The name of the configured similarity metric (e.g. "cosine").

Parameters:

  • backbone (str)

  • embedding_dim (int)

  • similarity_func (str)

  • transform (Compose | None)

  • device (str | device)

  • pretrained_backbone (bool)

  • normalize (bool)

  • batch_size (int)

class pyvisim.neural_networks.losses.TripletLoss(margin=0.2, mining='semi_hard', squared=True)[source]

Bases: Module

Triplet loss with online triplet mining, proposed in Schroff, F., Kalenichenko, D., & Philbin, J. (2015). FaceNet: A Unified Embedding for Face Recognition and Clustering.

For a triplet of anchor a, positive p (same class) and negative n (different class), the loss is the hinge

L(a, p, n) = max(0, d(a, p) - d(a, n) + margin)

where d is the (optionally squared) Euclidean distance. Instead of receiving precomputed triplets, the loss mines them online from a labeled batch of embeddings, exactly as in FaceNet: every image in the batch acts as an anchor and its partners are picked from the same batch. Offline triplet selection is deliberately not supported.

Supported mining strategies:

  • "semi_hard" (FaceNet): for every positive pair (a, p), pick the closest negative that is still farther than the positive (d(a, p) < d(a, n)). If no such negative exists in the batch, fall back to the farthest negative of the anchor. The loss is averaged over all positive pairs.

  • "batch_hard": for every anchor, use only its farthest positive and its closest negative (Hermans et al., 2017). The loss is averaged over all anchors that have at least one positive and one negative.

  • "batch_all": use every valid triplet in the batch and average over the triplets that violate the margin. Averaging over all triplets instead would let the many trivially satisfied ones wash out the signal (Hermans et al., 2017).

NOTE

"batch_all" and "semi_hard" build a (batch, batch, batch) comparison tensor, so their memory cost grows cubically with the batch size; "batch_hard" stays quadratic.

A batch that yields no valid triplet (e.g. it contains only one class) produces a zero loss that is still connected to the autograd graph, so loss.backward() keeps working in the training loop.

References:

[1] Schroff, F., Kalenichenko, D., & Philbin, J. (2015). FaceNet: A Unified Embedding for Face Recognition and Clustering. CVPR. https://doi.org/10.1109/CVPR.2015.7298682

[2] Hoffer, E., & Ailon, N. (2014). Deep Metric Learning Using Triplet Network. https://arxiv.org/abs/1412.6622

[3] Hermans, A., Beyer, L., & Leibe, B. (2017). In Defense of the Triplet Loss for Person Re-Identification. https://arxiv.org/abs/1703.07737

param margin:

Margin enforced between positive and negative distances. FaceNet uses 0.2 for squared distances.

param mining:

Online mining strategy, one of "semi_hard", "batch_hard" or "batch_all".

param squared:

If True, use squared Euclidean distances as in FaceNet. Hermans et al. report better convergence with plain distances (False), typically combined with "batch_hard".

raises ValueError:

If margin is not strictly positive or mining is not a supported strategy.

forward(embeddings, labels)[source]

Computes the mined triplet loss over a labeled embedding batch.

Parameters:

  • embeddings (Tensor) – Embeddings of shape (batch, dim).

  • labels (Tensor) – Class labels of shape (batch,); any dtype supporting equality comparison (integers in practice).

Returns:

The scalar loss.

Raises:

ValueError – If embeddings is not 2-dimensional, if labels is not 1-dimensional, or if the two disagree on the batch size.

Return type:

Tensor

Parameters: