ClipEmbedder

Embeds images with a pretrained CLIP image tower. Embeddings are L2-normalized by default, which makes the cosine similarity a plain dot product.

from pyvisim.neural_networks import ClipEmbedder

embedder = ClipEmbedder("ViT-B-32", pretrained="laion2b_s34b_b79k")
embeddings = embedder.embed(images)                # (N, 512)
score = embedder.similarity_score(image1, image2)  # (1, 1) cosine similarity

Supported models

Variant names and pretrained tags follow open_clip. OpenAI-style spellings such as "ViT-B/32" are accepted as aliases of "ViT-B-32".

Variant	Embedding dim	Input size	Pretrained tags
RN50	1024	224x224	openai, yfcc15m, cc12m
RN50-quickgelu	1024	224x224	openai, yfcc15m, cc12m
RN101	512	224x224	openai, yfcc15m
RN101-quickgelu	512	224x224	openai, yfcc15m
RN50x4	640	288x288	openai
RN50x4-quickgelu	640	288x288	openai
RN50x16	768	384x384	openai
RN50x16-quickgelu	768	384x384	openai
RN50x64	1024	448x448	openai
RN50x64-quickgelu	1024	448x448	openai
ViT-B-32	512	224x224	openai, laion400m_e31, laion400m_e32, laion2b_e16, laion2b_s34b_b79k, datacomp_xl_s13b_b90k, metaclip_400m, metaclip_fullcc
ViT-B-32-quickgelu	512	224x224	openai, laion400m_e31, laion400m_e32, metaclip_400m, metaclip_fullcc
ViT-B-32-256	512	256x256	datacomp_s34b_b86k
ViT-B-16	512	224x224	openai, laion400m_e31, laion400m_e32, laion2b_s34b_b88k, metaclip_400m, metaclip_fullcc
ViT-B-16-quickgelu	512	224x224	openai, metaclip_400m, metaclip_fullcc
ViT-B-16-plus-240	640	240x240	laion400m_e31, laion400m_e32
ViT-L-14	768	224x224	openai, laion400m_e31, laion400m_e32, laion2b_s32b_b82k, commonpool_xl_s13b_b90k, metaclip_400m, metaclip_fullcc
ViT-L-14-quickgelu	768	224x224	openai, metaclip_400m, metaclip_fullcc
ViT-L-14-336	768	336x336	openai
ViT-L-14-336-quickgelu	768	336x336	openai
ViT-H-14	1024	224x224	laion2b_s32b_b79k, metaclip_fullcc, metaclip_altogether
ViT-H-14-quickgelu	1024	224x224	metaclip_fullcc
ViT-H-14-worldwide	1024	224x224	metaclip2_worldwide
ViT-H-14-worldwide-quickgelu	1024	224x224	metaclip2_worldwide
ViT-H-14-worldwide-378	1024	378x378	metaclip2_worldwide
ViT-g-14	1024	224x224	laion2b_s12b_b42k, laion2b_s34b_b88k
ViT-bigG-14	1280	224x224	laion2b_s39b_b160k, metaclip_fullcc
ViT-bigG-14-quickgelu	1280	224x224	metaclip_fullcc
ViT-bigG-14-worldwide	1280	224x224	metaclip2_worldwide
ViT-bigG-14-worldwide-378	1280	378x378	metaclip2_worldwide

The -quickgelu names are open_clip spellings kept for compatibility, not separate architectures: whether the tower uses the QuickGELU activation of the original OpenAI models or the exact GELU of newer checkpoints is read off the checkpoint itself. A variant and its -quickgelu twin therefore build the same model for every tag they share. The plain name only exists separately because some of them offer extra tags (for example ViT-B-32 adds the LAION and DataComp checkpoints).

To print all supported variants and pretrained tags, use these helper functions:

from pyvisim.neural_networks.clip import available_pretrained, available_variants

print(available_variants())              # every supported variant name
print(available_pretrained("ViT-B-32"))  # every pretrained tag of one variant

API reference

class pyvisim.neural_networks.ClipEmbedder(variant='ViT-B-32', pretrained='openai', *, device=None, normalize=True, similarity_func='cosine', cache_dir=None, batch_size=16)

Bases: SerializableImageEmbedder

Embeds images with a pretrained CLIP model.

The safetensors checkpoint of the requested variant and pretrained tag is downloaded from the Hugging Face Hub on first use and cached (see pyvisim.neural_networks.clip.fetch_checkpoint()); only the image tower is loaded, and it always runs in float32. embed() returns one embedding per image, L2-normalized when normalize is on, so they can be compared directly with a dot product or the cosine similarity metric.

Variant names and pretrained tags follow open_clip, e.g. ClipEmbedder("ViT-B-32", pretrained="laion2b_s34b_b79k"). OpenAI-style variant spellings ("ViT-B/32") are accepted as aliases. See pyvisim.neural_networks.clip.available_variants() and pyvisim.neural_networks.clip.available_pretrained() for the supported combinations.

Parameters:

• variant (str) – CLIP variant name.

• pretrained (str) – Pretrained tag naming the weights, e.g. "openai" for the original OpenAI checkpoint of the variant.

• device (str | None) – Device to run the model on ("cpu" or "cuda"). If None, "cuda" is used when a CUDA device is available, else "cpu". The model runs in float32 on either device.

• normalize (bool) – Whether to L2-normalize the returned embeddings.

• similarity_func (str) – Name of the built-in similarity metric used to score two embeddings. One of "cosine", "euclidean", "l1" or "manhattan".

• cache_dir (str | Path | None) – Directory of the Hugging Face Hub cache the checkpoint is stored in. If None, the standard Hub cache (~/.cache/huggingface/hub) is used, so weights already downloaded via open_clip’s Hub downloads are reused.

• batch_size (int) – Maximum number of images processed in a single batch. Set to -1 to process all images as a single batch.

Raises:

• ValueError – If variant, pretrained or similarity_func is not supported, or the checkpoint does not match the architecture.

• ImportError – If the nn extra is not installed.

• huggingface_hub.errors.HfHubHTTPError – If the checkpoint download fails.

References:

[1] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel

Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever, “Learning Transferable Visual Models From Natural Language Supervision,” in Proc. ICML, PMLR 139, pp. 8748-8763, 2021.

embed(images, *, dims='HWC', value_range=(0.0, 255.0))

Embeds one or more images into a batch of vector representations.

Each image is normalized to a canonical uint8 (H, W, C) array before feature extraction, so NumPy arrays, torch tensors and other array-like inputs are all accepted. When a batch axis is present (via dims), every image in the batch is embedded. The resulting vectors are L2-normalized row by row when normalize is True.

Parameters:

• images (_Buffer | _SupportsArray[dtype[Any]] | _NestedSequence[_SupportsArray[dtype[Any]]] | bool | int | float | complex | str | bytes | _NestedSequence[bool | int | float | complex | str | bytes] | Iterable[_Buffer | _SupportsArray[dtype[Any]] | _NestedSequence[_SupportsArray[dtype[Any]]] | bool | int | float | complex | str | bytes | _NestedSequence[bool | int | float | complex | str | bytes]]) – A single MatLike image, a batched array, or an iterable of images. Consider using an iterator for large datasets.

• dims (str) – Axis-label string, one character per array axis in order: "H" = height (rows), "W" = width (columns), "C" = channels (e.g. RGB), "B" = batch size. For example, "HWC" is height × width × channels (NumPy/OpenCV single-image layout); "CHW" is channels × height × width (PyTorch single-image layout); "BCHW" is batch × channels × height × width (PyTorch batched layout). See pyvisim.typing.

• value_range (tuple[float, float]) – The (low, high) range the input values live in; converted into the canonical [0, 255] range.

Returns:

vector representations of the given images, L2-normalized row by row if normalize is True.

Raises:

ValueError – If images holds no image.

Return type:

ndarray[tuple[int, …], dtype[floating[Any]]]

classmethod from_dict(state, **kwargs)

Rebuilds the embedder a state dictionary describes.

Called on SerializableImageEmbedder itself, it hands the state to the from_dict of the class named under "__class__", so a state can be rebuilt without knowing which embedder wrote it.

Parameters:

• state (dict[str, Any]) – A JSON-safe embedder description.

• kwargs (Any) – Objects the state cannot describe, forwarded to the embedder’s own from_dict.

Returns:

The reconstructed embedder.

Raises:

• ValueError – If state names no concrete embedder class.

• NotImplementedError – If called on a subclass that does not implement its own from_dict.

Return type:

ClipEmbedder

classmethod load_from_disk(path, **kwargs)

Loads an object previously saved with save_to_disk().

Not every part of an object survives serialization: an arbitrary callable such as a torchvision transform has no portable description, so it is left out of the file. Pass such an object back here as a keyword argument.

Parameters:

• path (str | Path) – Path to the file to load.

• kwargs (Any) – Objects the file cannot hold, forwarded to from_dict().

Returns:

A ready-to-use instance.

Raises:

• FileNotFoundError – If path does not exist.

• ValueError – If the file is not a valid file of this kind or was saved by a different class.

• TypeError – If the class does not take one of kwargs.

Return type:

_SerializableT

save_to_disk(path)

Saves the serialized state of this object to a file.

Parameters:

path (str | Path) – Target file path. Overwritten if it exists.

Returns:

The path of the written file.

Raises:

OSError – If the destination directory does not exist.

Return type:

Path

set_batch_size(batch_size)

Sets the number of items processed per batch.

Parameters:

batch_size (int) – Maximum number of images processed in a single batch. Set to -1 to process all images as a single batch.

Raises:

ValueError – If batch_size is neither -1 nor a positive integer.

Return type:

None

similarity_score(images1, images2, *, dims='HWC', value_range=(0.0, 255.0))

Compute the similarity scores matrix between two (batches of) images.

Parameters:

• images1 (_Buffer | _SupportsArray[dtype[Any]] | _NestedSequence[_SupportsArray[dtype[Any]]] | bool | int | float | complex | str | bytes | _NestedSequence[bool | int | float | complex | str | bytes] | Iterable[_Buffer | _SupportsArray[dtype[Any]] | _NestedSequence[_SupportsArray[dtype[Any]]] | bool | int | float | complex | str | bytes | _NestedSequence[bool | int | float | complex | str | bytes]]) – First (batch of) image(s) as MatLike (NumPy array, torch tensor or array-like).

• images2 (_Buffer | _SupportsArray[dtype[Any]] | _NestedSequence[_SupportsArray[dtype[Any]]] | bool | int | float | complex | str | bytes | _NestedSequence[bool | int | float | complex | str | bytes] | Iterable[_Buffer | _SupportsArray[dtype[Any]] | _NestedSequence[_SupportsArray[dtype[Any]]] | bool | int | float | complex | str | bytes | _NestedSequence[bool | int | float | complex | str | bytes]]) – Second (batch of) image(s) as MatLike.

• dims (str) – Axis-label string, one character per array axis in order: "H" = height (rows), "W" = width (columns), "C" = channels (e.g. RGB), "B" = batch size. For example, "HWC" is height × width × channels (NumPy/OpenCV single-image layout); "CHW" is channels × height × width (PyTorch single-image layout); "BCHW" is batch × channels × height × width (PyTorch batched layout). See pyvisim.typing.

• value_range (tuple[float, float]) – The (low, high) range the input values live in; converted into the canonical [0, 255] range.

Returns:

The similarity score matrix of shape (len(images1), len(images2)).

Return type:

ndarray[tuple[int, …], dtype[float32]]

to_dict()

Serializes this object into a JSON-safe state dictionary.

The mapping holds the output of _state() plus the format version under "format_version" and the class name under "__class__". Arrays may be embedded as __ndarray__ nodes, which the serialization layer stores as binary tensors.

Returns:

A JSON-safe description suitable for from_dict().

Return type:

dict[str, Any]

property embedding_dim: int

Dimensionality of the embeddings produced by embed().

property image_size: int

Side length in pixels of the square model input.

property normalize: bool

Whether the embeddings returned by embed() are L2-normalized.

property pretrained: str

The pretrained tag naming the loaded weights (e.g. "openai").

property similarity_func: Callable[[ndarray[tuple[int, ...], dtype[floating[Any]]], ndarray[tuple[int, ...], dtype[floating[Any]]]], ndarray[tuple[int, ...], dtype[floating[Any]]]]

The resolved similarity function callable.

property similarity_func_name: str

The name of the configured similarity metric (e.g. "cosine").

property variant: str

The CLIP variant name in open_clip spelling (e.g. "ViT-B-32").