Annotations on a Budget: Leveraging Geo-Data Similarity to Balance Model Performance and Annotation Cost

Abstract: Current foundation models have shown impressive performance across various tasks. However, several studies have revealed that these models are not effective for everyone due to the imbalanced geographical and economic representation of the data used in the training process. Most of this data comes from Western countries, leading to poor results for underrepresented countries. To address this issue, more data needs to be collected from these countries, but the cost of annotation can be a significant bottleneck. In this paper, we propose methods to identify the data to be annotated to balance model performance and annotation costs. Our approach first involves finding the countries with images of topics (objects and actions) most visually distinct from those already in the training datasets used by current large vision-language foundation models. Next, we identify countries with higher visual similarity for these topics and show that using data from these countries to supplement the training data improves model performance and reduces annotation costs. The resulting lists of countries and corresponding topics are made available at https://github.com/MichiganNLP/visual_diversity_budget.

• The paper demonstrates a novel approach using geo-data similarity to identify underrepresented (topic, country) pairs for targeted annotation.
• It shows how annotating visually similar data groups can improve model accuracy while reducing overall annotation costs.
• The study delivers actionable insights for creating inclusive AI systems by balancing dataset diversity with enhanced performance.

Leveraging Geo-Data Similarity to Enhance Model Performance with Cost-effective Annotations

Introduction to Geo-Diverse Data Collection Challenges

The recent advancements in vision-LLMs have harnessed the power of large datasets to achieve state-of-the-art performance across various tasks. Yet, the inherent geographical and economic imbalances in these datasets have raised concerns regarding their inclusivity and effectiveness globally. Particularly, the underrepresentation of data from non-Western and low-income countries undermines the models' performance on data from these regions. Addressing this imbalance necessitates an increase in geo-diverse data collection. However, the high costs associated with data annotation pose significant challenges. This research proposes a novel approach to identify data for annotation that efficiently balances model performance improvement with annotation costs.

Identifying Underrepresented Geographical Data

To determine which countries are less represented in vision-LLM training data, this study first identifies images of topics that are visually distinct across various countries compared to those prevalent in Western-centric datasets. By analyzing image data across 52 countries and 94 topics, the study classifies (topic, country) pairs less represented in high-resource datasets, pointing towards where annotation efforts could be most beneficial.

Utilizing Geo-Data Similarity for Cost-effective Annotations

A core contribution of this paper is demonstrating how leveraging visual similarity across countries can significantly optimize annotation efforts. For (topic, country) pairs found to be underrepresented, the study identifies visually similar country groups for these topics. Annotating data from countries within these groups is shown to improve model performance while reducing the need for extensive annotations across all underrepresented domains. This approach not only highlights a path towards more balanced datasets but also provides practical insights into managing annotation costs without compromising on the diversity of the data.

Implications for Future AI Developments

The findings of this research are profound, suggesting that the path to developing more inclusive vision-LLMs lies in strategic data collection and annotation efforts. By identifying specific areas where data is lacking and leveraging similarities across geographies, researchers can create more balanced datasets that enhance model performance across diverse geographical contexts. Moreover, this method opens avenues for further research on dataset diversity, such as exploring other factors (e.g., cultural, economic) that contribute to visual data similarity.

Speculations on Future Developments

Looking ahead, it is likely that the strategies outlined in this paper will guide the development of more cost-effective and diverse datasets. These datasets, in turn, could fuel the next generation of AI systems that are truly global in their understanding and application. Furthermore, exploring additional dimensions of data similarity, beyond geography, could enhance the efficiency of data augmentation techniques, making it possible to achieve broad representation in AI models without proportional increases in annotation costs.

Conclusion

This study presents a crucial step towards addressing the imbalance in geographical representation within vision-LLMs' training data. Through innovative methods for identifying underrepresented data and leveraging cross-country visual similarities, this research proposes a cost-effective roadmap towards more inclusive and effective AI models. The implications of this work extend beyond immediate dataset enhancement strategies, provoking a broader consideration of how we approach the challenge of creating truly global AI systems.