Identifying Biases in VLMs' Zero-Shot Classifications of Race and Gender
An audit framework comparing how vision-language models label race and gender in face images against annotations from a demographically balanced sample of U.S. adults.
Do Vision Language Models’ (VLMs) zero-shot classifications of race and gender reflect the viewpoint of a specific demographic subgroup? To address this question, we propose a novel audit framework that compares the labeling of human face images across different VLMs with annotations from a demographically balanced sample of U.S. adults comprising two racial groups (White and Black) and two gender groups (male and female). This design will allow us to identify which social group’s classifications most closely resemble each model’s outputs.
Leads: Silvia Téliz Martínez, Cristina Monzer
Team: Deen Freelon
The project is supported by a $10,000 USD Annenberg School for Communication Internal Grant, with Silvia Téliz as principal investigator, for the 2026–2027 academic year.