Implicit Bias in Algorithms: The Fruit Doesn’t Fall Far From The Tree
A recent report by Freedom House, a human rights advocacy group, shows that 16 governments have already used generative AI to control what’s being published on the internet “to sow doubt, smear opponents, or influence public debate”. Some of these countries are Pakistan, Nigeria, and the US.
With the spread of generative AI, questions of the design behind the technology progressively occupy public discussions. As they should.
I recently came across an AdWeek article that expressed concerns about existing ethnical bias in the AI tool Mid Journey that’s used to generate images based on a textual prompt.
An agency asked Mid Journey to create an image of a Hispanic or Latinx man working. What they got back was quite concerning — the AI seemed stuck on stereotypes, churning out images of a guy with a big mustache and a sombrero.
Similarly, whenever they prompted Mid Journey to produce a picture of a woman, they would often get results of a blonde, blue-eyed female. No need to explain the obvious bias towards Western cultural standards.
One of the most accurate methods of measuring levels of unconscious bias in people is through the Implicit Association Test (IAT). In a typical test, participants face a computer with two buttons labeled ‘good’ and ‘bad’, along with two photos — one of a white man and one of a black man.
In the first scenario, participants were asked to press ‘good’ whenever they see the white man, and ‘bad’ whenever they see the black man. Most of them didn’t give it too much thought. However, in the second stage of the experiment, it was found that people genuinely take twice as long to press the button when they have to associate ‘black’ with ‘good’ and ‘white’ with ‘bad’.
And now, what happens when these preferences are embedded into the core of technology that governments progressively start to use?
In the following lines, you’ll find a discussion on bias in technology.
Along with a list of ways in which people influence AI while building it, I’ve provided a brief overview of a framework to overcome computer bias by Batya Friedman and Helen Nissenbaum written in 1996 (!!).
It’s astonishing how much knowledge of technology people had back then (even in the 1940s, but that’s a story for some other time), so I thought to share it here and hopefully spark ideas about how to properly implement it.
Bias in technology exists way before GenAI
Between 2015–2020, when people started to realise that basic algorithms are directing the majority of their online interactions, I remember there were many scandals around bias found in various systems.
Some of the most influential tech companies were accused of demonstrating ethnical, sexual and financial biases towards their employees, clients, and potential new applicants.
Here are a few examples from ‘back in the days’:
- In 2015, Google Photos faced controversy when its image recognition algorithm classified a black couple as ‘gorillas.’ This glaring racial bias demonstrated the challenges of training algorithms to recognise diverse faces accurately.
- In 2019, the Apple Card, issued by Goldman Sachs, faced allegations of gender bias. It was reported that the credit limit offered to women was often significantly lower than that offered to men, even in cases where the women had better credit scores.
- Amazon scrapped an AI-based hiring tool in 2018 after it was revealed that the algorithm was biased against women. The system had learned from resumes submitted over a decade, which were predominantly from men, leading the AI to favour male candidates.
Bias in algorithms developed by tech giants like Apple, Google, and Amazon often originates from the training data used to build and fine-tune these algorithms.
Algorithms, in their own right, are like silent machines waiting for data to bring them to life. Without datasets and trainings, algorithms are pretty useless.
I have identified 5 ways in which people can influence the emergence of bias in algorithms:
- Data selection bias: Training datasets can sometimes be skewed or fail to represent the rich diversity of the population they intend to serve. For instance, if the Apple Card training data predominantly consisted of mostly men with high income, it’s obvious why the algorithm would favour men.
- Historical biases: Training data often mirrors historical biases and prejudices that persist in society. This encompasses long-standing stereotypes, cultural biases, and historical injustices that may be deeply ingrained in the text, images, or other data used for training, much like what probably happened with Mid Journey.
- Human labelling bias: During training sessions, algorithms learn from data that has labels, much like contextual categories. You can already see how the process of labelling or annotating data can introduce bias → it’s humans who classify the labels after all.
- Feedback loops: Tech companies’ algorithms frequently engage in continuous learning and adaptation based on user interactions and feedback. If the initial algorithm exhibits bias, user feedback and interactions will magnify these biases over time, establishing a feedback loop that further embeds the bias.
- Implicit bias in user behaviour: User behaviour on tech platforms can also contribute to algorithmic bias. For instance, if users tend to click on or engage more with specific types of content, algorithms may prioritise and recommend similar content, potentially amplifying existing biases in user preferences.
(The list is by no means complete, and I’d love to hear your suggestions as well.)
Many of the points above include the human factor because bias in technology ultimately comes from the people who create and train that technology.
It’s a bit sad to think that there’s no actual way for us to change that — so much has been built on top of these systems in the last 10 years, that it’s hard to reverse the damage.
The only way for us to tackle implicit bias in algorithms seems to be to create patterns for inclusion in those systems. In order to do that, we first have to analyse which areas need more variety in data, and where bias is expressed.
Framework for identifying bias in AI
The article that I’m going to reference is from 1996, and is called “Bias in Computer Systems” by Batya Friedman and Helen Nissenbaum.
Back then, the authors called it “computer bias”, but the definition behind the term is practically equivalent:
We use the term bias to refer to computer systems that
systematically and unfairly discriminate against certain individuals or
groups of individuals in favor of others. A system discriminates unfairly if
it denies an opportunity or a good or if it assigns an undesirable outcome to
an individual or group of individuals on grounds that are unreasonable or
inappropriate.
Friedman and Nissenbaun analysed complex computer systems used in healthcare, banking and institutions, and found that bias existed in number of public spheres, including The National Resident Match Program (a centralised method for assigning medical school graduates their first employment following graduation), The British Nationality Act Program (used for issuing citizenships).
In their research, they identified and explored 3 categories of computer bias: preexesting bias, technical bias, and emergent bias.
Let’s see what each means:
- Preexisting Bias — rooted in societal institutions, practices, and attitudes, preexisting bias enters systems either explicitly or implicitly. It may originate from societal biases, subcultures, or the personal biases of those designing the system.
- Technical Bias — arising from limitations in technical design, technical bias can be found in hardware, software, algorithmic decisions, and attempts to translate human constructs into computational models.
- Emergent Bias — this bias unfolds over time due to shifting societal knowledge, demographics, or cultural values. Emergent bias is particularly likely to surface in user interfaces when the context of use changes.
You can see how applicable they still are. Even though technology itself has evolved, the tools we use to interpret it has not. So, if we take this matrix, we can analyse basically every technology out there.
For the purpose of this article, I’m going to create a very top-level application of this framework.
I’ll be trivial and try with Instagram.
- Preexisting Bias — the pre-existing bias in Instagram would be related to the cultural and societal background of its creators, in this case — Western, white, heterosexual, male (just assuming here).
- Technical Bias — Instagram’s technical bias would be that it’s algorithm is optimised towards clicks and engagement: the platform rates popularity of posts based on activity, and not actual content. This could lead to recirculating scammy, potentially false information only because it’s more clickable.
- Emergent Bias — That’s a tricky one, as we’re continuously in this phase, but an emergent bias could be that people start having preferences towards certain topics, appearances, and lifestyles, just because they are popular on Instagram. This is an example of an online behaviour influencing your ‘offline’ life.
While writing this text, I started thinking that this is a simple, yet effective thought exercise that prompts critical thinking, and could be of use when considering any kind of artificial intelligence tool.
Governments, financial structures, healthcare systems, and education would all benefit following Friedman and Nissenbaum’s the three-step analysis when considering various softwares. You don’t have to reinvent the wheel.
In our rapidly changing world driven by AI, we face significant challenges and responsibilities.
We’ve seen how algorithms can carry hidden biases and how critical training data is in shaping AI systems.
The development and regulation of AI systems are currently within the grasp of only a handful of big companies with vast resources, while many governments are still figuring out how to even form rules around AI.
As we witness both the immense potential for AI’s benefits and its potential for harm, policymakers stand at a crossroads. Their decisions in the coming years will define whether AI becomes a tool for fairness, equality, and progress, or if it perpetuates biases and inequalities…
I want to end on a positive note.
In August 2023, the EU passed final iterations for the Artificial Intelligence Act that would (most importantly) ban unacceptable risk technologies.
EU defines unacceptable risk technologies as softwares that: i) aim to manipulate, observe and analyse the emotions, behaviour and cognition of people; ii) classify people based on status, or ‘social scoring’, and iii) monitor biometric data in real time, including facial recognition.
Whew!!!
We have to build and adapt technology based on the collective thought of sociologists, historians, lawyers, anthropologists, psychologists, philosophers, engineers. And politicians and software developers should start allowing that.
It is a pivotal moment for democracies worldwide, calling for collaborative efforts between government, industry, and civil society to ensure that the principles guiding AI are not merely ideals but enforceable laws that protect our fundamental rights and values.
