From BLM to IBM: Racism and Bias in AI

Media Technology and Society Media Technology and Society Media Technology and Society

From BLM to IBM: Racism and Bias in AI

With IBM deciding to withdraw from all facial recognition business, we want to explore a major problem in AI: racism and bias.

A post by Mika Engelhardt and Johannes Eckes

Tuesday, 10 November 2020

Earlier this year (2020), the tech giant IBM announced that the company would shut down all its facial recognition business. At the same time, CEO Arvind Krishna called for a “national dialogue” on whether it should be used at all.
The timing of this may or may not have been intentional, but it all took place against the backdrop of the Black Lives Matter protests that erupted following the death of George Floyd in the US and around the world, and which are still ongoing to this day.

Apart from the fact that IBM was not making any significant profit from facial recognition, another reason for the change was bias in the systems themselves. Specifically, this means that the accuracy of facial recognition applications can vary depending on characteristics such as age, gender or race. This finding is not particularly new, as it was already demonstrated in studies from 2018 and 2019, yet IBM is the first major player in the tech sector to respond to this in a meaningful way.

How can AI become racist?

However questionable the use of facial recognition may be, it has still improved significantly over the last few years. At the same time, however, the technology has been shown to suffer from biases which can make the tools unreliable and even downright racist or sexist. But how does that happen? A company as large as IBM would surely never sell an openly racist algorithm, as the inevitable PR crisis would likely be catastrophic. Even less so do we wish to accuse any developers of harbouring racist intentions, or of reflecting these in their work.

Upon closer examination of the problem, it becomes clear that the bias usually lies not in the algorithm itself, but in the data on which it relies. Any artificial intelligence can only be as good as the data it is provided with. Put simply, this means that artificial intelligence systems learn from experience and feedback, much like children. There are various training methods in use at present, but the majority of them rely on the AI starting with assumptions about how to solve a task and receiving feedback on the outcome – either from a user or another programme. The feedback is then sent back to the AI in the form of a score and incorporated into the next analysis. The AI repeats this process, attempting to maximise the score with each iteration. This cycle continues until the score reaches its maximum.

But what happens if a facial recognition system is only fed images of one ethnic group or only of men? In this case, it is highly likely that a person of a different ethnic group or gender will be misclassified or even fail to be recognised at all. This is because, unlike humans, an AI is only concerned with the raw pixel values of an image. It does not perceive any deeper meaning in an image beyond the shapes, colours and textures it is searching for. As a result, a person with a different skin colour to the people in the dataset used is unlikely to be identified.

Other examples

With IBM taking this very public step away from artificial intelligence, we can also look at other instances in which AI has been shown to make decisions or behave in a racist manner.
One of the most famous examples is ‘Tay’, an artificial intelligence bot created by Microsoft and launched on Twitter on 23 March 2016. ‘Tay’ was intended to demonstrate how artificial intelligence can learn from everyday life, but after just 16 hours, Microsoft had to shut it down and delete the account after ‘Tay’ began tweeting racist and misogynistic content. Over 96,000 tweets were posted by “Tay” during this short period, and whilst at first they only concerned celebrities or horoscopes, this quickly shifted towards controversial topics such as the Holocaust (“Tay” posted tweets praising Hitler and saying she “hated Jews”), racist statements or 9/11. This was mainly because “Tay” was fed by online trolls who influenced her behaviour to such an extent that it became irreparable. A second version of “Tay” had to be taken offline even more quickly. This is a special case, because we can link the AI’s racist behaviour to trolling and influence by humans who were either racist themselves or treated it as a joke, but it remains a noteworthy example of just how fragile and difficult AI behaviour can be.
Another example that is far more damning for people is the US healthcare system, which also uses AI. In 2019, it emerged that the software implemented deeply racist behaviour. In hospitals, a software programme decides which patients qualify for a special programme that deals with long-term issues relating to diabetes. The AI was fed data on the costs incurred by all patients to the healthcare system, but disregarded individual needs or high-risk patients. As a result, people of African-American heritage were downgraded and did not qualify for the special treatment, because they (generally) use the healthcare system less frequently than white people. This can be traced back to structural problems in the USA. People of colour are still disadvantaged; they often have less money or live in more deprived neighbourhoods than white people. White people, however, can afford to visit the doctor more often and, as a result, generate more data for the AI to learn from. Ironically, this behaviour not only harms Black people but also the US healthcare system, because the costs of follow-up treatment for long-term effects are much higher than the actual treatment the AI was intended to assign to people.
Many hospitals and institutions in the USA use this or similar software, and it is said that the disadvantages for people of colour are severe.

Possible solutions

So, ultimately, the question remains as to whether anything can be done about these problems. We cannot simply tell a machine to be ‘more ethical’ or ‘less racist’, as it is just that – a machine. It has no concept of human ethics, which is in itself incredibly complex and often based on feelings, not on facts (data).
But there are solutions to this problem.

Firstly, whilst ethics are complex and often highly subjective, they can be broken down into simple sets of rules, which in turn can be programmed into an algorithm. This idea is not new, and its most prominent example is the three laws of robotics by the science fiction author Isaac Asimov. Another, more recent application is the ongoing debate about self-driving cars and their decision-making in critical situations (e.g. hitting a pedestrian versus saving the driver). In this case, various solutions have already been proposed, but no official decision has yet been made.
Traditionally, societies have relied on governments to ensure ethical standards are upheld through legislation and policing. Consequently, it is these governments that are, and will be, the ones compelled to take action as AI becomes increasingly widespread. This process is already underway, with the European Commission, the USA, the OECD and several other NGOs having established committees or drawn up roadmaps on the subject in recent years.

Secondly, and probably most obviously, a human oversight body to supervise decisions made by an autonomous system is another option for improvement. However, this is only a temporary solution, as an autonomous machine controlled by a human is not truly autonomous and thus contradicts the central idea of AI. But until another, safe solution is found, this may be the best option for keeping AI in check.

Lastly, sufficiently diverse datasets will be necessary to ensure the sustainable and non-discriminatory use of AI. Ultimately, however, the biases that emerge in AI are merely a reflection of the biases afflicting society. With this in mind, it becomes clear that this issue cannot be addressed without tackling systemic and societal bias. However abstract and mysterious AI may seem to the individual, the problems that affect it are simply the same as those that affect us all.

From BLM to IBM: Racism and Bias in AI

How can AI become racist?

Other examples

Possible solutions

Further reading and recommendations: