How to keep your (EdTech) algorithms honest (Part I — sharing your labour)
Why product managers are essential to safeguarding algorithmic bias
Algorithms are not born agnostic. They are designed and implemented via a series of human choices. Conscious or not, we inject bias and even prejudice into the algorithmic products we create.
In EdTech, there is a growing emphasis on efficacy, which holds products accountable to the educational outcomes they claim to improve. We must impose just as much scrutiny on the product design process. Too often, algorithmic products are presented as black boxes, riddled with hidden biases that discriminate against the users they purportedly serve. Unchecked algorithms have a knack for widening inequalities; some even assume racist behaviours.
Transparency has to be the an edict of EdTech, the core value that all stakeholders — users, buyers and creators — unite behind. So then: how can we hold our algorithms to account? How do we prevent them from automating and amplifying the worst of our prejudices?
How do we tear apart the black box and fulfil this pledge of transparency?
In this series of posts I will share some practical steps for safeguarding your algorithms from bias. We start with sharing your labour.
EdTech algorithms are often lost in translation. The following tale, while fictional, may sound painfully familiar:
A new educational product is dreamt up by a well-intended CEO. Said CEO has a grand vision of scaling high quality instruction through technology and, in doing so, reducing the achievement gap. His rough idea is an adaptive learning tool, a digital instructor of sorts, that delivers content based on each student’s predicted learning patterns — individualised learning pathways for each student.
The CEO does not have the technical skill to develop a prototype, nor the appetite to detail his thinking. Instead he communicates his idea of a digital instructor as a set of high-level requirements, before handing them over to the equally well-intended delivery team of engineers and data scientists.
The delivery team gathers fragments of the CEO’s thinking. It is not long before they turn to familiar machine learning models that they think will achieve the CEO’s aims. Grabbing a corpus of historical data, the delivery team proceeds to develop a neural network that learns from historical responses to predict how students will perform in the future.
No model can emulate exactly what the CEO is after, but he is impressed by the apparent sophistication of the model, and by the accuracy measures that seem to validate it. While he doesn’t really understand each layer of the model, or even have grasp of the assumptions on which it is based, the CEO is satisfied that the digital instructor behaves as required, and authorises the release to market.
The product goes live and all seems well until, after a while, the machines awake. No wait, that’s Hollywood. Rather, a curious pattern emerges: it seems that the digital instructor is widening the knowledge gap between students. It has predicted great things for the high attainers and is stretching them with deeper content. The low attainers, however, have their future written in stone: since the model that the digital instructor is based on predicts that students will not attain academic excellence in the long run, it restricts their learning path to a narrow band of topics, keeping them forever entrapped in low-level content.
Both the CEO and the delivery team are perplexed: neither had budgeted for this phenomenon — quite the opposite, in fact. Fingers are pointed in both directions. The CEO is accused of a lack of clarity, the engineers told they have made lazy assumptions. And someone, somewhere is shaking their head wondering why the hell they expected anything different.
That someone is most likely a product manager, whose role exists to mediate between business stakeholders, developers and end users, and to align the intended behaviours of products with the end user experience. In short, they are custodians of the product requirements.
With algorithmic products of any kind, the scope for misinterpreted requirements cannot be understated. No algorithm can be developed with objective truth, including — and especially — those underpinned by Machine Learning. Every statistical model contains assumptions, such as which variables to include as predictors, and trade-offs, such as the relative importance of precision and recall. Every decision reflects an approximation of our ideal product behaviours. We may select a seemingly neutral variable like postcode, failing to realise that it serves as a proxy for ethnic minority users. And as we add more complexity to our model, we may lose our grip on all notions of accuracy.
In our vignette, the model is accurate with respect to historical data but discriminatory when projecting forward. Put differently, the algorithm does no better than to perpetuate historical inequalities. Even in cases where we are satisfied that a model ‘mostly’ gets it right, black box algorithms such as neural networks expose us to a host of unintended consequences when errors are made. The one person who may have had the contextual insight to circumvent these issues is the CEO, who is also the one person who is unable to engage with the nitty gritty of machine learning.
For all of these reasons, a central Product team, comprised of a healthy mix of commercial, technical and design expertise, is needed to interrogate every assumption, and to check that the nuts and bolts of algorithmic products reflect the principles upon which they were developed.
The generous view of EdTech is that algorithmic bias is unintentional. That hardly makes it justified. Separating engineers from key stakeholders is an unconscionable oversight with predictable outcomes.
Do not rely on a non-technical stakeholder to fully define algorithmic product requirements and, likewise, never leave the interpretation solely to engineers who are removed from the context in which those algorithms are applied. Do pay your dues and commit to the integrated Product team your users deserve.
Next up in this series: even with the right team in place, how do we overcome the tendency of machine learning models to perpetuate historical bias?
