[DRAFT] Same Concept, Different Directions
Recover features first. Align them afterward.
Mechanistic interpretability studies a model’s internal computations. The linear representation hypothesis proposes that concepts correspond to directions in representation space. [3]
A sparse autoencoder (SAE) learns feature directions whose sparse combinations reconstruct embeddings—the model’s vector representations. [4]
IMAGEA jet in the sky
TEXT“A jet is flying
through the sky.”
Same concept. Must image and text features share a direction?
SAE reconstruction
Each activation weights a decoder column. Summing these contributions and the decoder bias reconstructs the embedding.
Paper Figure 7
From an image to its reconstruction
Select a concept and photograph.
01 · INPUT
Input image
1 / 5
VLM embedding: x.
02 · SPARSE FEATURES
Latent coordinate
Baseball
Image latent 1789
Activation z̃j(x) weights decoder column wj.
03 · RECONSTRUCTION
Sparse reconstruction
+ decoder bias
Activation × decoder column = one feature contribution.
Cross-modal feature heterogeneity
Cross-modal feature heterogeneity: corresponding image and text features can have different directions.
Compare reconstruction and latent alignment
Select a representation and vary the angle.
One shared decoder
(1, 0)(0, 1)Per input · lower is better
Exact reconstruction; different latent coordinates (modality split).
Analytic example · §5
Limited capacity: which features merge?
Reduce the decoder columns; change the feature labels.
Features per decoder column
Select a decoder column.
Feature recovery, then post-hoc alignment
- 1
Feature recovery
Train modality-specific SAEs to reconstruct image and text embeddings.
- 2
Post-hoc alignment
Match latent coordinates using activation correlations on paired inputs.
Reorder coordinates. Preserve reconstruction.
Corresponding features occupy different coordinates.
Text activations
Reconstructed embedding (toy)
Each activation remains paired with its decoder column.
Activation correlation matrix
Hungarian matching maximizes total activation correlation, excluding coordinates that never activate. Correlation does not guarantee semantic correspondence.
Reconstruction invariance
P permutes decoder columns and activations together; the decoder bias is unchanged.
Reconstruction and cross-modal retrieval
Paper Table 1
Modality-specific SAEs: before and after alignment
Reconstruction MSE
MS-COCO MSE · lower is better
Cross-modal retrieval
Image → text Recall@1 · higher is better
Feature heterogeneity · Four VLM families
Feature recovery · Synthetic experiments
Concept-conditioned image retrieval
Feature recovery first. Post-hoc alignment preserves reconstruction.
References
- Same Concept, Different Directions: Cross-Modal Feature Heterogeneity in Sparse Autoencoders · Chungpa Lee, Jihoon Kwon, Kyle Min, and Jy-yong Sohn.
- Learning Transferable Visual Models From Natural Language Supervision · Alec Radford et al. · ICML 2021.
- The Linear Representation Hypothesis and the Geometry of Large Language Models · Kiho Park, Yo Joong Choe, and Victor Veitch · ICML 2024.
- Towards Monosemanticity: Decomposing Language Models With Dictionary Learning · Transformer Circuits · 2023.