← Posts

[DRAFT] The Optimal Geometry of Contrastive Learning Objectives

Contrastive losses, batch partitions, and class labels favor different embedding geometries.[1][2][3]

Contrastive learning

An encoder maps each input to a vector, called an embedding.[4][5]

Data inputsE.g. image, text
EncoderE.g. ResNet, ViT
EmbeddingsNumerical vectors

Positive pairs are treated as matches; negative pairs as non-matches.

Augmentation

Dog photograph in Khosla et al., Figure 5(b)
The same source dog photograph, rotated and cropped for this augmentation example
Two views of one image
Khosla et al. (2020), Fig. 5(b), adapted

Image–caption pairs

Original puppy image from Radford et al., Figure 1
Original caption: Pepper the aussie pup
An image and its caption
Radford et al. (2021), Fig. 1, cropped

Class labels

Dog photograph in Khosla et al., Figure 5(b)
dog
A different dog photograph in Khosla et al., Figure 5(c)
dog
Different images, same label
Khosla et al. (2020), Fig. 5(b–c), cropped

Four instances, eight embeddings

Four instances, two views each: uᵢ and vᵢ form a positive pair.

Vectors have unit norm. Cosine similarity is 1 for identical directions, 0 for orthogonal directions, and −1 for opposite directions.

The matrix has four positive diagonal entries and twelve negative off-diagonal entries.[2]

Eight embeddings

(Cosine) similarity matrix

Mean similarity of positive pairs
Mean similarity of negative pairs
Choose an arrangement; select a matrix entry to identify the pair.

Optimization over embeddings

We minimize losses over unit embeddings in sufficient dimension. A trained encoder may impose additional constraints.

Softmax vs. sigmoid contrastive losses

InfoNCE normalizes across candidates; sigmoid loss assigns a penalty to each pair.[7][6]

Here n=3: uᵢ–vᵢ are positive pairs; uᵢ–vⱼ, i≠j, are negative pairs.

Symmetric InfoNCE averages six anchor losses; sigmoid sums three positive and six negative pair terms.[4]

Similarities are scaled by 1/τ.[1][6]

Select a loss term to highlight its similarities. Bias b shifts the sigmoid scores.

Sigmoid terms can be summed independently. InfoNCE requires each complete row or column denominator.[6]

Alignment and separation

For n=3, setting every negative similarity to −1 also forces every positive similarity to −1.

With aligned positives, the minimum negative mean is −1/(n−1), attained by a simplex equiangular tight frame (ETF): an equilateral triangle for n=3.[1][2]

Minimize negative similarity

Maximize positive similarity

0.55
One pointSimplex ETF
The slider changes spacing between instances while positive pairs remain aligned.

Double-Constant Embedding Model

The double-Constant Embedding Model (CCEM) attains the minimum negative mean for each fixed positive mean.

All positive pairs share one similarity; all negative pairs share another.

For image–text pairs, separation between view centers corresponds to a modality gap.

Negative similarity =Positive similarity − 34
0.6
−11

For unit embeddings with d≥n, τ>0, and fixed b≥0, CCEM contains a global minimizer of sigmoid loss (Corollary 2).[1]

Temperature and optimal geometry

InfoNCE is minimized by aligned simplex embeddings at every temperature.

Sigmoid can favor misaligned positives. With b=1 here, alignment begins at τ≈0.185.[1]

0.140
0.0050.200
The dot marks the minimum loss; the geometry shows its optimizer.

Here t=1/τ and b=1; the paper’s Theorem 2 sets t=b.[1]

Two families of contrastive losses

Two loss families, three choices of negative pairs.[2]

InfoNCE-based losses

Compare positive and negative similarities.

ψ(∑j=23φ((𝒗j−𝒗1)⊤𝒖1))\psi\!\left(\sum_{j=2}^{3}\phi\!\left((\boldsymbol v_j-\boldsymbol v_1)^\top\boldsymbol u_1\right)\right)

Examples: InfoNCE, SimCLR.

Independently additive losses

Add positive and negative penalties separately.

−φ(𝒖1⊤𝒗1)+12∑j=23ψ(𝒖1⊤𝒗j)-\phi(\boldsymbol u_1^\top\boldsymbol v_1)+\frac12\sum_{j=2}^{3}\psi(\boldsymbol u_1^\top\boldsymbol v_j)

Example: sigmoid loss.

One anchor u₁; cross-view negatives v₂ and v₃. Each family defines its own φ and ψ.[2]

Which pairs does the loss compare?

Cross-view: uᵢ–vⱼ. Within-view: uᵢ–uⱼ or vᵢ–vⱼ, i≠j. Toggle comparisons and vary δ.

The same three pairs, different comparisons

0.50
Aligned · δ = 0Separated · δ = 1
Embedding geometry
Similarity matrix
PositiveCross-view negativeWithin-view negative— Not included

Select a point or a matrix entry to inspect a pair.

Positive pairs0.60At alignment: 1.00Included
Cross-view negatives−0.60At alignment: −0.50Included
Within-view negatives−0.20At alignment: −0.50Not penalized

Illustrative CCEM configurations, not a training trajectory. Means include pairs omitted from the loss.[2]
Full-batch optimum
Loss familyCross-view onlyWithin-view onlyBoth
InfoNCE-basedAligned simplexAligned simplexAligned simplex
Independently additiveDepends on attraction and repulsionAligned simplexAligned simplex

Aligned simplex: positive similarity 1, negative similarity −1/(n−1). Assumptions: Theorems 5.1 and 5.3.[2]

A limit independent of the loss

A mean cross-view negative similarity below −1/(n−1) rules out perfect positive alignment for any normalized encoder in this setup (Theorem 5.2).[2]

For n=3, the threshold is −0.50. Above it, alignment is not guaranteed.
Definitions and proofs
Empirical validation

Numerical optimization

CCEM prediction and reproduced optimization results for ten instances. Mean positive similarity increases from minus one to one as t increases with b=t. Optimization points overlap the theoretical curve.
Reproduced optimization of 10 instances with t=b=1/τ. Points average three random initializations.[1]
Experiment details

10-dimensional embeddings; 50,000 projected-gradient steps; step size 0.5.

Adapted from author implementation (§5.2). This reproduction sets t=b=1/τ.

Full-batch vs. mini-batch training

Fixed mini-batches omit comparisons between batches, leaving their relative geometry undetermined.[2]

Two batches can move independently

Four instances; fixed batches {1,2} and {3,4}. Each two-instance InfoNCE optimum aligns the views at two antipodal points.[7]

Rotating either batch preserves its minimum loss. Moving to the tetrahedron attains the full-batch minimum and increases the fixed mini-batch loss.

Statistics include all u–v comparisons, including pairs omitted from the loss.[2]

Expectation = mean

Variance = mean squared distance from the mean

1
Coincident linesTetrahedron

Embedding geometry

Similarity matrix

Expectation and variance

Start at Coincident lines. At Orthogonal lines, release and drag again. Uncheck Reveal all pairs to hide omitted comparisons during rotation.

Same mean, different geometry

Negative mean stays −1/3; negative variance falls from 8/9 through 2/9 to 0.

Positive pairs remain aligned: mean 1, variance 0.

For fixed disjoint batches, both variance bounds decrease with batch size and vanish at full batch (Theorem 5.5).[2]

An auxiliary loss penalizes squared deviations from −1/(n−1), where n is the dataset size.[2]

Definitions and proofs
Empirical validation

Variance regularization

Reported negative-pair similarity variance at batch sizes 32, 64, 128, 256 and 512. SimCLR values are 0.1649, 0.1505, 0.1444, 0.1404, 0.1396. With the variance penalty they are 0.1008, 0.0952, 0.0929, 0.0921, 0.0917. The penalty reduces variance at every tested batch size.
ResNet-18 on CIFAR-100, trained with randomly sampled mini-batches. Values reported in Table 1.[2]

Negative-similarity variance decreases; downstream accuracy depends on the experimental setting.[2]

Values and experiment details

5,000 augmented pairs; SimCLR temperature 0.2; penalty weight 30. Experiments use random batches; Theorem 5.5 assumes fixed partitions.

Variance of negative-pair similarities
Batch sizeSimCLR+ variance penalty
320.16490.1008
640.15050.0952
1280.14440.0929
2560.14040.0921
5120.13960.0917

Self-supervised vs. supervised losses

Class collapse occurs when all instances in a class share one embedding.[3]

Instance matches and class matches

Four instances, two views each; classes {1,2} and {3,4}.[3]

ℒself attracts views of the same instance; ℒsup attracts different instances in the same class.

Same-instance pairs appear only in ℒsup’s denominator, following the paper’s decomposition.

Select an anchor to inspect its positive and denominator-only pairs.

Both denominators include all eight embeddings. Fixed self-comparisons are omitted from the drawing.
ℒα = (1 − α)ℒsup + αℒself

Increasing α favors instance separation.[3]

Simplex-to-Simplex Embedding Model

The Simplex-to-Simplex Embedding Model (SSEM) describes the global optima: views align, while within-class spread varies.

Within-class variance 0 means class collapse; 2/3 gives a tetrahedron.[3]

Similarities in the four-instance SSEM
PairSimilarity
Two views of the same instance1
Different instances, same class1 − 2 × within-class variance
Different classes−1 + within-class variance

For this balanced, normalized setting with d≥3, minimizing the one-parameter SSEM curve yields a global optimum.

At τ=0.5, collapse persists through α≈0.518. At α=1, the optimum is a tetrahedron.[3]

0.4
SupervisedSelf-supervised
The dot marks the optimal within-class variance; zero means class collapse.

Geometric measurements

  • Positive similarity: alignment.
  • Negative mean and variance: separation and its variability.
  • Within-class variance: class collapse.
Definitions and proofs
Empirical validation

Numerical optimization

Within-class variance versus self-supervised weight alpha at temperatures 0.5 and 1.0. Blue SSEM predictions and reproduced optimization points agree. The class-collapse region has zero variance and is wider at temperature one.
Reproduced optimization of four instances with two views each. Points average three random initializations; shading marks class collapse.[3]
Experiment details

Eight embeddings in three dimensions; Adam, 2,000 steps, learning rate 0.5; unit normalization at each forward pass.

Adapted from author implementation (§6.1). Points use optimized embeddings; curves use SSEM.