Evidence of Limited Biases in Probed Language-based Assessments
Sparhuber, M., Eijsbroek, V., Kjell, K., Giorgi, S., & Kjell, O. (2026). Evidence of Limited Biases in Probed Language-based Assessments. PsyArXiv. https://doi.org/10.31234/osf.io/zrb3v_v1
@misc{sparhuber2026evidence,
title = {{Evidence of Limited Biases in Probed Language-based Assessments}},
author = {Marc Sparhuber and Veerle Eijsbroek and Katarina Kjell and Salvatore Giorgi and Oscar Kjell},
year = {2026},
howpublished = {PsyArXiv},
doi = {10.31234/osf.io/zrb3v_v1},
url = {https://doi.org/10.31234/osf.io/zrb3v_v1}
}
Abstract
Large language models (LLMs) enable increasingly accurate language-based assessments (LBAs) of psychopathology. As these tools are more widely deployed in real-world settings, a critical question emerges: Do these tools work equally well for everyone? We examined whether probed LBAs, in which individuals respond to open-ended mental health questions in their own words, exhibit systematic biases that could undermine their clinical utility. We collected open-ended mental health descriptions from 1,260 participants alongside validated depression (PHQ-9) and anxiety (GAD-7) scores. We converted these descriptions to numeric representations using LLM-generated embeddings and applied ridge regression to predict symptom severity, achieving strong predictive accuracy (r > .60). We then examined whether assessment differences were systematically associated with sociodemographic factors (gender, age, education, employment, income) or linguistic characteristics. Our analysis revealed limited evidence of demographic or linguistic bias: most variables showed minimal association with assessment errors (i.e., the absolute differences between the LBAs and self reports). However, higher depression and anxiety severity were associated with larger absolute assessment differences (standardized β = .193, p = .038; β = .252, p = .003). We explored mitigation strategies, finding that downsampling training data eliminated the bias for depression and anxiety. Our findings indicate that probed language-based assessments exhibit limited bias overall, particularly regarding sociodemographic equity. However, the observed severity-related pattern highlights a fundamental challenge.
