← Posts

[DRAFT] Same Concept, Different Directions

Recover features first. Align them afterward.

Mechanistic interpretability studies a model’s internal computations. The linear representation hypothesis proposes that concepts correspond to directions in representation space. [3]

A sparse autoencoder (SAE) learns feature directions whose sparse combinations reconstruct embeddings—the model’s vector representations. [4]

IMAGEA jet in the sky

TEXT“A jet is flying
through the sky.”

Same concept. Must image and text features share a direction?

01

SAE reconstruction

Each activation weights a decoder column. Summing these contributions and the decoder bias reconstructs the embedding.

Paper Figure 7

From an image to its reconstruction

Select a concept and photograph.

01 · INPUT

Input image

A baseball player pitching, reproduced from Figure 7

1 / 5

VLM embedding: x.

02 · SPARSE FEATURES

Latent coordinate

Baseball

Image latent 1789

Activation z̃j(x) weights decoder column wj.

03 · RECONSTRUCTION

Sparse reconstruction

x̂ =z̃1789(x) w1789+ contributions from the other active features
+ decoder bias

Activation × decoder column = one feature contribution.

Photographs and latent indices: Figure 7. Reconstruction is schematic; individual activations are not reported. [1]
02

Cross-modal feature heterogeneity

Cross-modal feature heterogeneity: corresponding image and text features can have different directions.

Compare reconstruction and latent alignment

Select a representation and vary the angle.

Image featureText feature

One shared decoder

Shared decoder
Image code(1, 0)
Uses coordinate 1 → image direction
Shared decoder
Text code(0, 1)
Uses coordinate 2 → text direction
Squared reconstruction error
Per input · lower is better
0.000

Exact reconstruction; different latent coordinates (modality split).

60°
0° · Same direction80° · More different
Illustrative unit inputs with one active coordinate. Directions and correspondence are given; option 2 imposes one shared coordinate. Dotted gaps show reconstruction errors.

Analytic example · §5

Limited capacity: which features merge?

Reduce the decoder columns; change the feature labels.

3
1 · Most constrained4 · One per direction
ImageTextSelected decoder column

Features per decoder column

Select a decoder column.

Mean squared error0.0085
Merged coordinates1

Exact calculations for Top-1 reconstruction of four unit directions; not training results. The eigenvector sign condition holds. See Theorem 1 for the general assumptions.
03

Feature recovery, then post-hoc alignment

  1. 1

    Feature recovery

    Train modality-specific SAEs to reconstruct image and text embeddings.

  2. 2

    Post-hoc alignment

    Match latent coordinates using activation correlations on paired inputs.

Reorder coordinates. Preserve reconstruction.

Image decoderText decoder

Corresponding features occupy different coordinates.

Text activations

(0.00, 0.70, 0.50)

Reconstructed embedding (toy)

(0.573, 1.043)

Each activation remains paired with its decoder column.

Illustrative directions, labels, and correlations. Matching reorders coordinates; it does not recover missing directions.
Activation correlation matrix

Hungarian matching maximizes total activation correlation, excluding coordinates that never activate. Correlation does not guarantee semantic correspondence.

Reconstruction invariance
(WP)(Pᵀz) = Wz

P permutes decoder columns and activations together; the decoder bias is unchanged.

04

Reconstruction and cross-modal retrieval

Paper Table 1

Modality-specific SAEs: before and after alignment

Reconstruction MSE

MS-COCO MSE · lower is better

Cross-modal retrieval

Image → text Recall@1 · higher is better

CLIP ViT-B/32 · CC3M → MS-COCO · K = 32 · Three-run averages. Recall@1: correct first-ranked matches (%). Endpoints show the same SAEs before and after alignment. Table 1.
Feature heterogeneity · Four VLM families
Distributions of cosine distances between image and text feature directions in CLIP, MetaCLIP, OpenCLIP, and SigLIP2.
Cosine distance between paired feature directions; zero means identical directions. c: activation correlation. Paper Figure 2.
Mean cosine distances between independently trained SAEs, for feature pairs with activation correlation ≥ 0.6. Paper Table 2(a).
Feature recovery · Synthetic experiments
Synthetic reconstruction and feature recovery errors as the distance between image and text feature directions varies.
Synthetic data; equal total parameter counts. Modality-specific SAEs preserve close, distinct directions; the shared SAE performs better when directions coincide. Paper Figure 4.
Alignment and feature recovery errors under increasing auxiliary-loss weights, compared with separate SAEs and post-hoc alignment.
Feature distance = 0.5. Stronger auxiliary losses can degrade feature recovery; post-hoc alignment preserves it. Paper Figure 5.
Concept-conditioned image retrieval
Concept-conditioned image retrieval: an elephant example and mAP and MRR comparisons for three alignment methods.
Match a text feature to an image feature, then shift the image query along that decoder column. mAP and MRR: 80 COCO categories. Paper Figure 6.

Feature recovery first. Post-hoc alignment preserves reconstruction.

References
  1. Same Concept, Different Directions: Cross-Modal Feature Heterogeneity in Sparse Autoencoders · Chungpa Lee, Jihoon Kwon, Kyle Min, and Jy-yong Sohn.
  2. Learning Transferable Visual Models From Natural Language Supervision · Alec Radford et al. · ICML 2021.
  3. The Linear Representation Hypothesis and the Geometry of Large Language Models · Kiho Park, Yo Joong Choe, and Victor Veitch · ICML 2024.
  4. Towards Monosemanticity: Decomposing Language Models With Dictionary Learning · Transformer Circuits · 2023.