DEADEYE: Differential Expressivity As Dataset fairnEss/usabilitY Estimator

Abstract

Over the past several years, significant research has gone into analyzing algorithmic fairness -- the problem of ensuring ML algorithms do not exhibit biases against protected groups. That research demonstrated that, given a fair ground truth dataset, one could produce algorithms that maintained that fairness (for various definitions of fairness). Additionally, that research gave several holistic ways in which datasets themselves might be unfair. We provide a new metric for dataset fairness, \textit{Differential Expressivity}, which puts dataset fairness on the same formal grounding as algorithmic fairness. Additionally, we show several hardness results for this new metric, as well as algorithms for calculating it in certain subcases. Finally, we test the metric on COMPAS recidivism data and show that empirically it points out underlying fairness issues on real data.