When Does Perceptual Alignment Benefit Vision Representations?

Abstract: Humans judge perceptual similarity according to diverse visual attributes, including scene layout, subject location, and camera pose. Existing vision models understand a wide range of semantic abstractions but improperly weigh these attributes and thus make inferences misaligned with human perception. While vision representations have previously benefited from alignment in contexts like image generation, the utility of perceptually aligned representations in more general-purpose settings remains unclear. Here, we investigate how aligning vision model representations to human perceptual judgments impacts their usability across diverse computer vision tasks. We finetune state-of-the-art models on human similarity judgments for image triplets and evaluate them across standard vision benchmarks. We find that aligning models to perceptual judgments yields representations that improve upon the original backbones across many downstream tasks, including counting, segmentation, depth estimation, instance retrieval, and retrieval-augmented generation. In addition, we find that performance is widely preserved on other tasks, including specialized out-of-distribution domains such as in medical imaging and 3D environment frames. Our results suggest that injecting an inductive bias about human perceptual knowledge into vision models can contribute to better representations.

• The paper introduces a novel hinge loss strategy to align model outputs with human judgments, improving perceptual coherence.
• It demonstrates that human-aligned models achieve better results in dense prediction and retrieval tasks without sacrificing generalization.
• The study leverages the NIGHTS dataset and patch-level propagation techniques to optimize vision representations via human perceptual insights.

Analyzing Perceptual Alignment in Vision Representations

The paper "When Does Perceptual Alignment Benefit Vision Representations?" investigates the impact of aligning vision model representations with human perceptual judgments. It critically examines the utility of such alignment across various computer vision tasks, providing a nuanced understanding of its implications.

Overview

The research addresses the longstanding challenge that while vision models understand a range of semantic abstractions, they often misalign with human perceptual assessments. Traditional vision models weigh attributes differently than humans, influencing the quality of inferences they make. The paper explores whether injecting an inductive bias about human perceptual knowledge can enhance these models, particularly when applied to downstream tasks such as counting, segmentation, depth estimation, and retrieval.

Methodological Approach

The study leverages the NIGHTS dataset, consisting of synthetic image triplets annotated with human similarity judgments. By finetuning state-of-the-art models like CLIP, DINO, and SynCLR on these judgments, the paper evaluates their performance across standard vision benchmarks.

• Alignment Loss: The paper employs a hinge loss to align model outputs with human judgments, minimizing the cosine distance between similar image representations and maximizing dissimilar ones.
• Patch-level Propagation: The research introduces a novel approach by propagating global human annotations to ViT patch tokens, optimizing for local features in the context of dense prediction tasks.

Key Findings

1. Performance Enhancement: Human-aligned models exhibit improved performance on several downstream tasks, achieving better results in dense predictions (e.g., depth estimation and segmentation) and retrieval-based tasks.
2. Generalization: The alignment does not significantly degrade performance in areas where models already excel, indicating strong generalization capabilities.
3. Retrieval-Augmented Generation (RAG): Models aligned to human perceptual judgments demonstrate enhanced abilities in retrieval-augmented generation tasks, potentially improving few-shot classification and retrieval tasks in vision-LLMs.
4. Sensitivity to Dataset Characteristics: The paper's dataset ablation studies highlight that mid-level perceptual judgments, such as those in NIGHTS, lead to significant performance improvements compared to low or high-level variations, which can detract from model utility in certain cases.

Implications and Future Directions

The implications of this study are twofold:

• Theoretical: It contributes to understanding how perceptual alignment can imbue models with capabilities that mimic human-like visual processing, influencing how models are trained and evaluated in the future.
• Practical: The improved performance in diverse tasks suggests practical applications in areas requiring nuanced visual discrimination, like autonomous vehicles and robotics.

Future research could explore extending perceptual alignment to other modality tasks and further explore optimal dataset characteristics for enhancing alignment benefits. Moreover, understanding the balance of alignment to maintain general-purpose effectiveness without compromising existing strengths is crucial.

Conclusion

This paper offers a significant contribution to the field by demonstrating that careful perceptual alignment can bolster model performance across various vision tasks. It provides a comprehensive analysis of when and how these alignments are beneficial, paving the way for future explorations in enhancing vision representations with human-centric perspectives.