Spotting and avoiding bias in AI

Why AI is not neutral

AI systems appear objective, but they are not. They are a co-construction of society and technology: they learn from data that people created, and they reproduce the inequalities of the world that data comes from. Anyone who uses AI therefore takes on not just a tool, but also the assumptions built into it.

Three well-known cases make this tangible. Amazon’s recruiting AI disadvantaged applications from women in 2018 because it had learned from ten years of predominantly male hiring data; the system projected the past onto the future and perpetuated the imbalance. Apple Health launched in 2014 claiming to track the relevant body functions, yet initially had no concept of a menstrual cycle; that feature was added only later. And the Gender Shades study by Joy Buolamwini and Timnit Gebru showed in 2018 that commercial facial classification from IBM, Microsoft, and Face++ had an error rate of up to 34.7 percent for darker-skinned women, compared with only 0.8 percent for lighter-skinned men.¹

The effect of these findings is notable. After Gender Shades made the numbers public, several of the tested vendors improved and significantly narrowed the error gap, especially for darker-skinned women; at the same time, the study fueled the policy debate over the use of facial recognition. This shows that bias is not a law of nature. Where it is measured and named, it can be reduced.

These cases share a pattern: no malicious intent, but data and processes that carry existing inequality forward invisibly. To recognize and prevent such effects systematically, Prof. Dr. Nicola Marsden at Heilbronn University developed a framework that shows where you need to intervene.⁵

How does the cycle of discrimination arise?

Marsden describes bias not as a single error, but as a cycle across four levels that reinforce each other:

• Product: biased data and algorithms that produce discriminatory results.
• Society: stereotypes and role models that flow into the data and are confirmed by the results.
• People: homogeneous teams with blind spots; the share of women in German IT is below 20 percent.
• Processes: development methods with no fixed fairness checks.

The cycle closes because each level feeds the next. Biased data produces discriminatory products. These products normalize societal stereotypes. Homogeneous teams without diverse processes do not even notice the problem, because they lack the perspective to see it. Anyone who intervenes at only one point, for example only in the data, is turning a wheel that the other three simply turn back.

You can see this in the Amazon case. At the product level, the algorithm learned from biased hiring data. That data was shaped by society, because the tech industry had been male-dominated for years. The developing team was presumably not very diverse itself and considered the data basis unproblematic. And the process included no fairness check that would have caught the error early. Only once the bias became obvious did Amazon withdraw the tool. All four levels were involved, and precisely for that reason a fix would have had to address all four levels too.

Where exactly does bias show up?

It helps to distinguish three levels of effect. Bias operates cognitively in the form of associations (for example the silent equation “IT is male”), emotionally in the form of reservations, and behaviorally in concrete conduct, meaning decisions and wording. The crucial point: discrimination usually does not happen through malicious intent, but through unconscious associations that nobody actively chooses, which is exactly why they are so hard to notice.

With AI, a second effect is added, automation bias: people tend to trust a machine output more than their own judgment, simply because it came from a computer. As long as it is a movie recommendation, that is harmless. But scaled across many consequential decisions, from applications to loan approvals to medical recommendations, it becomes a structural risk. That is exactly why the EU AI Act explicitly requires high-risk applications to check data for bias and minimize it. Fairness is thus no longer only an ethical question, but a regulatory obligation.

Why this pays off for businesses

Before turning to solutions, it is worth looking at the costs. Bias is not only a moral problem but a concrete operational one. A recruiting system that filters out strong candidates shrinks the talent pool and makes objectively worse decisions. A credit or diagnostic system that disadvantages certain groups creates legal risk, especially under the EU AI Act, and lastingly damages customer trust once the case becomes public. Fairness is therefore not a luxury you add on top, but a component of quality. A biased model is simply a worse model, regardless of how impressive its metrics look at first glance.

Why structures matter more than convictions

The most common response to bias is an appeal: people should become aware of their prejudices and pull themselves together. That sounds reasonable, but it hardly works. Unconscious associations cannot be switched off by willpower.

A classic example shows what helps instead. US symphony orchestras were almost entirely male until into the 1970s. Only once auditions took place behind a curtain, so the jury could no longer see the candidates, did the share of women rise, in some analyses only once the footsteps on the way to the podium could no longer be heard either (Goldin and Rouse, 2000; the exact effect size is still debated methodologically). The lesson holds even if you argue about the numbers: structures work, appeals to attitudes barely do.

This is exactly where behavioral design comes in. The idea is not to try to train bias away, but to design processes so that the fair outcome becomes the easiest path, the default, not the exception. US legal scholar Joan C. Williams turned this idea into a measurable method under the name Bias Interrupters: you measure a concrete bias, build in a small structural interruption, and measure again. In documented cases, this reduced women’s disadvantage in access to career-advancing assignments from 13 percentage points to zero, and structured evaluation forms reduced racial bias in hiring by about a third.⁴ Williams treats bias the way an engineer treats a defect: instead of urging people to be more careful, you redesign the system so the error is harder to make.

For working with AI, two such structures can be applied right away, one for the team, one for how you handle the model.

Practice 1: Round Robin, hear every voice

In typical AI project meetings, a few loud voices dominate. Important perspectives, for example from subject-matter experts, end users, or compliance, do not make it into the decision, even though they are exactly the ones that would reveal blind spots. Round Robin is the simplest countermeasure: each person speaks in turn, without interruption, with equal speaking time.⁶

The mechanism behind it is well understood. When everyone speaks in sequence and undisturbed, conformity pressure drops, groupthink is broken up, and the team’s collective intelligence actually comes into play, instead of being drowned out by the three loudest people. A discussion that would otherwise quickly run along familiar lines becomes a process that systematically gathers new perspectives.

The method runs in four steps:

1. Set the stage: briefly explain the method, in turn, without interruption.
2. Visualize the topic: formulate a clear, focused question and make it visible.
3. Set the order: by seating order, alphabetically, or by a name list (works virtually too).
4. Moderate and summarize: capture points, name commonalities and differences.

The technique is especially helpful for requirements workshops, discussions about data selection, review meetings, and ethics rounds, exactly the places where the early course is set for a fair AI product. The effort is minimal, a few minutes of structure per meeting, and the payoff lies not only in more fairness but simply in better decisions, because more relevant knowledge reaches the table.

Practice 2: Debiasing through prompting

Large language models reproduce stereotypes by default. A widely cited study made this especially vivid: in 2023, Wan and colleagues had a language model write reference letters, changing only the first name of the person described. The result carries the title of the study: “Kelly is a Warm Person, Joseph is a Role Model.” The male-named candidate was more often described with achievement- and status-related terms, the female-named candidate more often with terms of warmth and likability.² Same performance, different name, different letter.

The good news: language models carry explicit knowledge about stereotypes within them, which can be activated through targeted prompting and turned against the bias.³ You can ask the model to check its own output. In practice, this works well in four steps, illustrated here with a reference letter.

Step 1, initial prompt:

Step 2, bias analysis:

Step 3, propose alternatives:

Step 4, rewrite:

Try it yourself with a real text from your daily work. You will be surprised what turns up in the first pass, and how much more factual and fair the rewritten version sounds.

An honest caveat belongs here too: debiasing prompting fixes the visible symptoms, not the cause. The model stays the same; only its specific output gets checked and smoothed. For everyday use it is still valuable, because it intervenes exactly at the point where the text actually gets used. What matters is not to skip the step: an AI text taken over unchecked carries the bias forward unfiltered. Anyone who makes the check a fixed habit turns a known weakness of the models into a controlled work step.

What you can take from this

Bias in AI is not a niche topic for ethics committees, it is a question of quality. A system that systematically disadvantages certain groups makes worse decisions, loses trust, and can become a legal problem in regulated areas. The reassuring message: you do not need to re-educate people to fix it. It is enough to build small, reliable structures into the right places.

The combination works best: Round Robin ensures that all perspectives on the team are heard before any AI even gets built, and debiasing prompting checks the outputs once the model is at work. Further practices such as diversity personas, a team manifesto, or fixed review rounds build on this; Prof. Nicola Marsden presents them in her free course on KI-Campus.⁵ The first step, though, costs nothing but a little structure, and you can take it in your very next meeting.