Biased Programmers? Or Biased Data? A Field Experiment in Operationalizing AI Ethics

Why does "algorithmic bias" occur? The two most frequently cited reasons are "biased programmers" and "biased training data." We quantify the effects of these using a field experiment on a diverse group of AI practitioners. In our experiment, machine learning programmers are asked to predict math literacy scores for a representative sample of OECD residents. One group is given perfectly representative training data, and the other is given a "dataset of convenience" -- a biased training sample containing who confirm to common expectations about who is good at math. Using this field experiment, we quantify the benefits of employing programmers who are diversity-aware vs obtaining more representative training data. We also measure the effectiveness of training interventions to reduce algorithmic bias, including both reminders and technical guidance.