AI Bias: What It Is and How to Prevent It | Built In

This post was originally published on this site.

AI bias occurs when an artificial intelligence system makes imbalanced or unfair decisions. These systems learn from large datasets, and if that data contains societal stereotypes or historical inequalities, the AI will absorb and repeat those patterns — resulting in biased outputs that have real-world consequences.

The harms of AI bias can be significant, especially in areas where fairness matters. A biased hiring algorithm may overly favor male applicants, inadvertently reducing women’s chances of landing a job. Or an automated lending tool may overcharge Black customers, hindering their chances of buying a home. And as artificial intelligence becomes more embedded in consequential industries like recruitment, finance, healthcare and law enforcement, the risks of AI bias continue to escalate.

“Whether we like it or not, all of our lives are being impacted by AI today, and there’s going to be more of it tomorrow,” Senthil Kumar, chief technology officer at data analytics company Slate Technologies, told Built In. “Decision systems are being handed off to machines — and those machines are biased inherently, which impacts all our lives.”

What Is AI Bias?

AI bias, also called machine learning bias or algorithmic bias, refers to the unfair decisions made by AI systems, caused by skewed data, flawed algorithms and inherent human biases. It is one of the greatest dangers of artificial intelligence because it not only mirrors real-world prejudices but also amplifies them — disproportionately favoring or discriminating against specific groups in ways that can perpetuate systemic inequality. 

What Causes AI Bias?

Bias primarily creeps into AI systems through training data. AI models learn how to make decisions and predictions based on the data they are trained on — and if that data is full of societal inequalities and stereotypes, those biases will inevitably be absorbed by the model and reflected in its outputs. Plus, if the data is incomplete or not representative of the broader population, the AI may struggle to produce fair and accurate results in scenarios it hasn’t encountered, further perpetuating discrimination.

“AI bias is born from data bias,” Adnan Masood, chief AI architect at IT services company UST, told Built In. “It is a mirror and magnifier of existing inequalities. It amplifies the prejudices we already have.”

Indeed, all artificial intelligence is the product of human beings, who are inherently biased, making it nearly impossible to avoid bias in AI systems. Developers may inadvertently introduce their own prejudices, overlooking important information while collecting data or teaching an algorithm to favor certain patterns during the machine learning process. In the end, the model will reflect those prejudices in its own decisions.

But unlike human decision-makers — whose biases can be more readily identified and challenged — AI systems operate in the background, often making decisions that are difficult (if not impossible) to fully understand or trust. This not only upholds existing inequalities but it also hinders adoption of the technology itself, as the public grows increasingly wary of systems they can’t fully count on or hold accountable.

Related ReadingWhat Is AI Safety?

What Are the Consequences of AI Bias?

Biased models are likely to spread their prejudices to the AI products they support, eventually impacting society. For example:

• If a large language model is taught to believe that all doctors are men and all nurses are women, its flawed logic will transfer over to the hiring tool it is powering, leading it to prioritize gender over qualifications in its decision-making.
• Facial recognition systems often don’t work as well for people with darker skin tones, which can lead to issues that range from something as minor as struggling to unlock a smartphone to more serious consequences like false arrests and wrongful deportations.
• According to UNESCO, the top AI chatbots have only been trained on about 1 percent of the world’s 7,000 natural languages (with English being the primary source), rendering them useless for large swaths of people.
• Even something as innocuous as recommendation systems in streaming services can exhibit bias, disproportionately promoting popular content in ways that prevent users from finding more diverse narratives they might also enjoy.

“Once a model has learned incorrect information, it is going to spread this knowledge,” Kumar said. “So the models aren’t just learning through biased datasets and biased processes, but they’re also propagating biases.”

Examples of Biased AI

AI bias is everywhere, influencing society in all kinds of ways. Here are some well-known examples:

Racial Bias in Generative AI

Generative AI tools — particularly image generators — have developed a reputation for reinforcing racial biases. The datasets used to train these systems often lack diversity, skewing towards images that depicted certain races in stereotypical ways or excluding marginalized groups altogether. As a result, these biases are reflected in AI-generated content, often portraying white people in roles of authority and affluence, and people of color as low-wage workers and criminals. Some experts predict that as much as 90 percent of online content will be artificially generated in the next few years, raising concerns that these biases will further perpetuate prejudice and hinder progress toward more equal representation.

Gender Bias in AI Voice Assistants

From Apple’s Siri to Amazon’s Alexa, many of today’s most popular assistants have feminine voices and embody traditionally feminine personas that inadvertently position them as subservient, submissive and overly obliging. This gendered design choice not only reflects harmful stereotypes, but it also influences how users interact with this technology — often leading to more dismissive, condescending and even harassing behavior toward the voice assistants. The ongoing feminization of voice assistants reinforces and upholds outdated notions of gender roles, carrying overly simplistic generalizations into modern times.

Sex and Racial Disparities in Healthcare Algorithms

Artificial intelligence is increasingly being applied in healthcare, from AI-powered clinical research to algorithms for image analysis and disease prediction. But these systems are often trained on incomplete or disproportional data, compounding existing inequalities in care and medical outcomes among specific races and sexes. For example, an algorithm for classifying images skin lesions was about half as accurate in diagnosing Black patients as it was white patients because it was trained on significantly fewer images of lesions on Black skin. Another algorithm developed to predict liver disease from blood tests was found to miss the disease in women twice as often as in men because it failed to account for the differences in how the disease appears between the sexes.

Ableism in AI Recruiting Tools

Hiring algorithms used to automatically screen applications have a demonstrated bias against individuals with disabilities — often because these systems are trained on data that only reflects able-bodied norms and assumptions. For example, programs that assess candidates’ communications skills, attention span, physical abilities and even facial expressions tend to unfairly disadvantage those with disabilities by failing to account for different modes of communication or movement. And resume scanners are apt to reject applicants with large gaps in their work history, without taking into account that those gaps may be due to health-related reasons. By reinforcing ableist hiring practices, AI recruiting tools limit job opportunities for individuals with disabilities and perpetuate discrimination in the job market at scale.

Class Bias in Automated Credit-Scoring

AI models for predicting credit scores have been shown to be less accurate for low-income individuals. This bias arises not necessarily from the algorithms themselves, but from the underlying data, which fails to accurately depict creditworthiness for borrowers with limited credit histories. A thin or short credit history can lower a person’s score because lenders prefer more data. It also means that just one or two small dings (a delinquent payment or a new credit inquiry) can cause outsized damage to a person’s score. As a result, these individuals may be wrongfully denied loans, credit cards and housing, which can then hinder their ability to build wealth and improve their economic situation down the road — perpetuating a cycle of financial disadvantage that further entrenches socioeconomic inequalities.

Related ReadingWhat Is Responsible AI?

Types of AI Bias

AI models can exhibit many types of AI bias, including:

• ​​Algorithmic bias occurs when the AI algorithm itself introduces or amplifies bias, often as a result of poor training data and the biases of the humans who compiled the data and trained the algorithm.
• Selection bias (also known as sample bias) arises when a training dataset is so small, disproportionate or incomplete that it fails to represent a real-world population, making it insufficient to properly train an AI model.
• Cognitive bias is introduced by the humans developing the model, stemming from their ingrained thought patterns and unconscious mental shortcuts that inadvertently shape how the data is interpreted and the AI system is designed. 
• Confirmation bias is closely related to cognitive bias; it involves humans favoring information that validates a pre-existing belief or trend. Even with accurate AI predictions, developers may ignore results that don’t align with their expectations.
• Historical bias occurs when outdated stereotypes and patterns of discrimination are embedded in the algorithm’s training dataset, causing the AI to replicate and reinforce these biases in new contexts. 
• Out-group homogeneity is the tendency to perceive members of a different group as more similar to each other than they actually are, often resulting in harmful stereotypes and generalizations. This can lead AI systems to misclassify individuals within that group, resulting in racial bias and inaccurate results.
• Exclusion bias is when important data is left out of a dataset, often because the developer has failed to understand its significance. 
• Recall bias happens during the labeling process, where data is inconsistently categorized based on the subjective observations of humans. 

How to Prevent AI Bias

Mitigating AI bias is a complex challenge that requires a multifaceted approach: 

Maintain a Diverse Development Team

Different perspectives can help identify potential biases early in the development stage. A more varied AI team — considering factors like race, gender, job role, economic background and education level — is better equipped to recognize and address biases effectively. 

“Homogenous teams are more likely to overlook biases,” Shomron Jacob, head of applied machine learning at AI development company Iterate.ai, told Built In. “Different people from different backgrounds bring in multiple perspectives. And those inputs end up mitigating biases by default, because they bring in a perspective other people wouldn’t consider.”

Train With the Right Data

Algorithms are only as good as the data they have been trained on, and those trained on biased or incomplete information will yield unfair and inaccurate results. To ensure this doesn’t happen, the training data must be comprehensive and representative of the population and problem in question.

“We should have diverse training data from a wide range of demographics and a wide range of scenarios. This is how you will reduce the likelihood of the system learning biased patterns,” Jacob said. “If you keep collecting bigger sets from different places, it will not learn any biased patterns directly.”

Implement Explainability Wherever Possible

The inner workings of AI models are often unclear, which makes it difficult to pinpoint the exact origins of their bias. Developers can opt for more explainable algorithms like LIME (Local Interpretable Model-agnostic Explanations) and SHAP (SHaply Additive exPlanations) to process the data, helping them to better scrutinize and evaluate their decision-making processes.

“If we have that traceability and explainability, that at least helps us to be able to further detect that this particular prediction might be biased, and we can take actions to prevent it,”Jim Olsen, chief technology officer at AI governance company ModelOp, told Built In. “Sometimes it’s not about not having any AI bias in the model itself, but it’s about the ability to detect it and stop it.”

Continually Monitor Models

AI models should be regularly monitored and tested for bias, even after they’ve been deployed. Models constantly take in new data with use and their performance can change over time, which may lead to new biases. Routine audits allow developers to identify and correct the issues they see before they cause harm.

“No model is ever going to be perfect, and that’s not the expectation. The expectation is that the due diligence is done,” Olsen said. “If you monitor your models constantly to make sure they aren’t drifting in their nature, their predictions, et cetera — and can prove when you are audited that you did all these steps — that’s all anybody can ask for.”

To further avoid bias, these assessments should be carried out by independent teams within the organization or a trusted third-party.

Establish More Regulations

Regulation can play an important role in addressing and mitigating AI bias by establishing guidelines and standards that ensure fairness and accountability. There are already many laws on the books protecting people from wrongful discrimination in areas like banking, housing and hiring (and several companies have been punished for violating those laws with AI). But for less obvious forms of AI bias, there are fewer legal safeguards in place.

Governments around the world have started taking steps to change that though, including the European Union, the United States and China. And various industry groups are implementing best practices in responsible AI development, promoting things like diverse data collection, transparency, inclusivity and accountability.

Related ReadingHow to Combat Bias in Machine Learning