by Javaid Sofi
Abstract
AI’s integration into healthcare, while promising enhanced diagnostics and efficiency, carries a significant risk of amplifying health disparities affecting vulnerable populations. Bias arises from unrepresentative datasets (e.g., dermatology algorithms trained on lighter skin tones), cognitive biases in clinical labelling, and flawed proxies like zip codes conflating geography with health risk. Studies like Obermeyer et al[1] demonstrate that recalibrating algorithms with equitable metrics can rectify disparities, proving bias is addressable. Mitigation requires inclusive data collection, transparent model design, and collaborative governance involving clinicians, policymakers, and communities. This blogpost aims to explore the ethical implications of bias in AI-driven healthcare and advocate for a multi-stakeholder approach to ensure equitable outcomes and prevent harm to vulnerable patient populations.
When Algorithms Decide Who Gets Care
At the heart of medical ethics is the principle of “Do No Harm”, along with principles of justice and equity, which emphasizes the responsibility to avoid causing harm to the patients. As the use of artificial intelligence increases and becomes integrated into healthcare, this principle has taken on a new relevance. Artificial intelligence (AI) holds promise in healthcare, offering rapid diagnostics, precision in treatment, and accuracy in predicting outbreaks. We define “AI-driven healthcare” as the application of computational algorithms, particularly machine learning, to analyze complex health data for tasks including diagnostics, treatment personalization, risk prediction, and operational efficiency.
AI-powered systems now can surpass human radiologists in speed and accuracy of medical image analysis[2] and play a crucial role in precision medicine, customizing treatments based on individual genetic profiles.[3] Researchers at National institute of Health created an AI tool capable of analyzing data from individual tumor cells to predict a cancer patient’s likelihood of responding to a particular drug which was demonstrated in a proof-of-concept study.[4]
AI offers scalable, cost-effective solutions, especially valuable in low-resource setting, in India these technologies are being used to improve TB management by forecasting treatment results, identifying at risk patients and boosting diagnostic precision through chest X-rays.[5] While eight in ten Americans think AI could improve healthcare quality,[6]this technology risks worsening existing health disparities.[7] Widespread AI adoption necessitates careful examination of algorithmic bias and proactive measures for equitable outcomes. Algorithms risk becoming tools of inequity, silently amplifying data-embedded biases[8]. Robust frameworks are crucial to mitigate potential harm and ensure equitable AI in healthcare. A stark illustration of why addressing this is critical emerged from the work of Obermeyer and colleagues.
The Obermeyer Study: Revealing Algorithmic Bias in Healthcare
Obermeyer and colleagues made a disturbing discovery: a healthcare algorithm used by hospitals across America was systematically biased against Black patients.[9] This single algorithm affected care decisions for approximately millions of Americans, quietly perpetuating racial disparities under the guise of objective data analysis. This wasn’t small-scale research finding, it revealed how a commercially deployed system was denying necessary care to thousands of Black patients nationwide. This did not constitute neutral metric; it suggested existing systematic inequalities. While the algorithm was not explicitly designed with racist intent, its reliance on past healthcare costs as a proxy for future health need reflected historical injustices in access and utilization. Consequently, the algorithm’s application led to the disparate allocation of healthcare resources, wherein fewer black patients received the support they needed.
What makes this case study particularly valuable is what happened next. When researchers replaced the flawed cost proxy with direct health measures, the racial disparity virtually disappeared. The percentage of Black patients identified for extra care dramatically increased from 17.7% to 46.5%.[10] This transformation demonstrates that algorithmic bias isn’t inevitable—it can be identified and remediated through thoughtful, equity-centered algorithm design. The “black box” nature of many AI algorithms compounds bias concerns. Their opacity makes it difficult to understand decision-making processes, raising ethical questions about accountability when biased algorithms lead to incorrect diagnoses.[11]
This discovery encapsulates why algorithmic bias in healthcare demands our urgent attention. When we delegate healthcare decisions to algorithms that silently encode societal biases, we risk automating discrimination at unprecedented scale and speed. The Obermeyer case powerfully demonstrates the impact of biased proxies, but this is just one of several ways systemic inequities can become embedded within healthcare algorithms.
How Bias Infiltrates Healthcare Algorithms: A Systematic Analysis
Machine learning AI identifies patterns in datasets. Biased or incomplete datasets, mirroring societal biases, can lead algorithms to replicate and amplify these biases, though mitigation strategies can lessen this effect. For example, an AI trained on skin cancer images primarily of lighter skin tones may struggle to accurately diagnose melanoma in darker skin, potentially delaying diagnoses and worsening outcomes.[12] This illustrates how dataset bias can significantly impact AI algorithms. A dataset of patients largely derived from those with access to healthcare will implicitly underrepresent communities facing barriers to healthcare. Algorithms trained on such datasets could inadvertently prioritize the health patterns of the overrepresented, potentially leading to misdiagnoses for the disadvantaged populations, or even deny services to them.[13]
Data labeling by medical professionals, whether conscious or unconscious, introduces bias and harmful feedback loops. Cognitive biases like confirmation bias anchoring bias, and availability heuristic can skew labeling. For example, confirmation bias might lead a radiologist to over-diagnose a high-risk patient even with inconclusive X-rays.[14] The use of proxy variables presents another challenge. When an algorithm uses accessible but often misleading data such as zip code as a stand-in for socioeconomic status, it conflates geography with health outcomes, reinforcing stereotypes and obscuring the root cause of disparities.[15] Individuals living in disadvantaged neighborhoods often experience higher rates of certain diseases, and while systematic factors such as access to healthcare and environmental conditions are significant drivers, there is often a complex interplay between biological and social determinants of health.
Moreover, the narrow geographic focus of training datasets, with many models relying disproportionately on data from a select few states like California, New York, and Massachusetts,[16] creates a significant barrier to widespread application because demographic and healthcare characteristics differ substantially from region to region. Algorithms developed predominantly using data from these specific states may struggle to make accurate and equitable predictions when applied to communities with different populations and social determinants of health.
Harnessing AI’s Potential for Equitable Healthcare: A Path Forward
While recognizing the inherent risks, artificial intelligence (AI) has the ability to transform healthcare delivery through its unique capabilities such as scalability which can facilitate population level health insights,[17] and continuous learning capacity which can ensure it evolves with medical advancement.[18] To harness this potential, it may be necessary to proactively address the limitations of AI by establishing robust processes for auditing and refining algorithms to minimize inequities over time. This requires a multifaceted approach that includes:
- Prioritizing Data Governance: Through inclusive data collection methods and multiple stages of rigorous quality checks,[19] we can ensure diverse and unbiased data sets. Inclusive data collection, however, immediately encounters the hurdle of patient privacy. Robust anonymization techniques are therefore key to navigating this tension.[20] Building trust through community ownership and engagement in data initiatives is essential for long-term success.
- Employing Robust Algorithm Design: By focusing on continuous improvements, transparency, and actively uncovering biases[21] we can strive to build fairer AI systems. However, algorithmic “black boxes” can undermine this very transparency. Therefore, prioritizing Explainable AI is crucial to illuminate decision-making processes.[22]
- Developing Ethical Framework: By ensuring that AI is developed and deployed with equity at its core, through collaborative partnerships with clinicians, data scientists, and diverse stakeholders. Furthermore, given the inherent potential for conflicting interests amongst these varied stakeholders, transparent ethical guidelines are critical. These guidelines must clearly define acceptable practices.
The transformative promise of AI in healthcare is undeniable, but it carries an equal potential to exacerbate health inequities if developed and deployed without careful consideration of bias. Bias can infiltrate AI systems at every stage, from data collection to algorithm design and clinical implementation. This underscores Melvin Kranzberg’s insight: “Technology is neither good nor bad; nor is it neutral.”[23] The impact of AI is shaped by human choices and values.
The story of healthcare algorithms need not be one of automated discrimination. The Obermeyer study demonstrates that when we critically examine our systems and center equity in their design, AI can help close rather than widen healthcare disparities. Their intervention reduced algorithmic bias not through complex technical means, but through the simple yet profound act of questioning what we measure and why.
It is our collective responsibility as researchers, developers, policymakers, clinicians, and institutions to proactively guide AI’s integration into healthcare, embedding principles of equity and justice. Only by confronting and mitigating bias can we unlock AI’s true potential to improve health outcomes for all, particularly for the vulnerable populations most at risk from unchecked technological advancement.
[1] Ziad Obermeyer and Sendhil Mullainathan, “Dissecting Racial Bias in an Algorithm That Guides Health Decisions for 70 Million People,” in Proceedings of the Conference on Fairness, Accountability, and Transparency (New York: Association for Computing Machinery, 2019), 89,https://doi.org/10.1145/3287560.3287593.
[2] Stanford Medicine News Center, “Artificial Intelligence Rivals Radiologists in Screening X-Rays for Certain Diseases,” Stanford Medicine, November 15, 2017, http://med.stanford.edu/news/all-news/2018/11/ai-outperformed-radiologists-in-screening-x-rays-for-certain-diseases.html
[3] Kevin B. Johnson et al., “Precision Medicine, AI, and the Future of Personalized Health Care,” Clinical and Translational Science 14, no. 1 (January 2021): 93, https://doi.org/10.1111/cts.12884.
[4] National Cancer Institute, “NIH Researchers Develop AI Tool with Potential to More Precisely Match Cancer Drugs to Patients,” news release, April 18, 2024, https://www.cancer.gov/news-events/press-releases/2024/ai-tool-matches-cancer-drugs-to-patients.
[5] SeshaSai Nath Chinagudaba et al., “Predictive Analysis of Tuberculosis Treatment Outcomes Using Machine Learning: A Karnataka TB Data Study at a Scale,” arXiv preprint arXiv:2403.08834, March 13, 2024, https://doi.org/10.48550/arXiv.2403.08834.
[6] Jesse Noyes, “Perceptions of AI in Healthcare: What Professionals and the Public Think,” The Intake (Tebra), last updated March 14, 2024, accessed April 20, 2025, https://www.tebra.com/theintake/medical-deep-dives/tips-and-trends/research-perceptions-of-ai-in-healthcare
[7] Irene Dankwa-Mullan, “Health Equity and Ethical Considerations in Using Artificial Intelligence in Public Health and Medicine,” Preventing Chronic Disease 21 (August 22, 2024): E64, https://doi.org/10.5888/pcd21.240245
[8] Zhisheng Chen, “Ethics and Discrimination in Artificial Intelligence-Enabled Recruitment Practices,” Humanities and Social Sciences Communications 10 (2023): 567, https://doi.org/10.1057/s41599-023-02079-x.
[9] Obermeyer and Mullainathan, “Dissecting Racial Bias,” 104
[10] Obermeyer and Mullainathan, “Dissecting Racial Bias,” 106.
[11] Matthew Kosinski, “What Is Black Box AI and How Does It Work?,” IBM Think, October 29, 2024, accessed April 20, 2025https://www.ibm.com/think/topics/black-box-ai. ibm.com
[12] Eman Rezk, Mohamed Eltorki, and Wael El-Dakhakhni, “Improving Skin Color Diversity in Cancer Detection: Deep Learning Approach,” JMIR Dermatology 5, no. 3 (August 19, 2022): e39143,pmc.ncbi.nlm.nih.gov
[13] Isabella Backman, “Eliminating Racial Bias in Health Care AI: Expert Panel Offers Guidelines,” Yale School of Medicine News, December 21, 2023, https://medicine.yale.edu/news-article/eliminating-racial-bias-in-health-care-ai-expert-panel-offers-guidelines/.
[14] Pat Croskerry, “Cognitive Forcing Strategies in Clinical Decisionmaking,” Annals of Emergency Medicine 41, no. 1 (January 2003): 115, https://doi.org/10.1067/mem.2003.22.
[15] Costanza Nardocci, “Proxy Discrimination in Artificial Intelligence: What We Know and What We Should Be Concerned About,” Chaire de recherche du Canada sur la culture collaborative en droit et politiques de la santé (blog), February 9, 2024, accessed April 20, 2025https://www.chairesante.ca/en/articles/2024/proxy-discrimination-in-artificial-intelligence-what-we-know-and-what-we-should-be-concerned-about/.
[16] Amit Kaushal, Russ Altman, and Curt Langlotz, “Geographic Distribution of US Cohorts Used to Train Deep Learning Algorithms,” JAMA 324, no. 12 (2020): 1212, https://doi.org/10.1001/jama.2020.12067.
[17] Angela Spatharou, Solveigh Hieronimus, and Jonathan Jenkins, Transforming Healthcare with AI: The Impact on the Workforce and Organizations, Executive Briefing (McKinsey & Company and EIT Health, March 10, 2020), https://www.mckinsey.com/industries/healthcare/our-insights/transforming-healthcare-with-ai
[18] Ophir Ronen, “Transparent AI In Healthcare: Transforming the Industry for the Better,” Forbes, December 5, 2023, accessed April 20, 2025, https://www.forbes.com/councils/forbestechcouncil/2023/12/05/transparent-ai-in-healthcare-transforming-the-industry-for-the-better/. councils.forbes.com
[19] Claudia Wells, “Inclusive Data to Leave No One Behind—Best Practices in Data Disaggregation and Use” (PDF file, UN Women, Commission on the Status of Women, 63rd Session, March 2019), https://www.unwomen.org/sites/default/files/Headquarters/Attachments/Sections/CSW/63/official-meetings/Claudia%20Wells%20updated.pdf
[20] Enlitic, “Deidentifying and Anonymizing Healthcare Data,” Enlitic (blog), January 25, 2023, accessed April 20, 2025,https://enlitic.com/blogs/deidentifying-and-anonymizing-healthcare-data/.
[21] David Jensen et al., “Algorithmic Robustness,” arXiv preprint arXiv:2311.06275, October 17, 2023, https://doi.org/10.48550/arXiv.2311.06275
[22] Stanford HAI, “Should AI Models Be Explainable? That Depends,” Stanford HAI News, March 16, 2021, accessed April 20, 2025,https://hai.stanford.edu/news/should-ai-models-be-explainable-depends.
[23] Melvin Kranzberg, “Technology and History: ‘Kranzberg’s Laws,’” Technology and Culture 27, no. 3 (1986): 545.
Bibliography
Backman, Isabella. “Eliminating Racial Bias in Health Care AI: Expert Panel Offers Guidelines.” Yale School of Medicine News, December 21, 2023. https://medicine.yale.edu/news-article/eliminating-racial-bias-in-health-care-ai-expert-panel-offers-guidelines/.
Chen, Zhisheng. “Ethics and Discrimination in Artificial Intelligence-Enabled Recruitment Practices.” Humanities and Social Sciences Communications 10 (2023): 567. https://doi.org/10.1057/s41599-023-02079-x.
Chinagudaba, SeshaSai Nath, Darshan Gera, Krishna Kiran Vamsi Dasu, Uma Shankar S, Kiran K, Anil Singarajpure, Shivayogappa U, Somashekar N, Vineet Kumar Chadda, and Sharath B. N. “Predictive Analysis of Tuberculosis Treatment Outcomes Using Machine Learning: A Karnataka TB Data Study at a Scale.” arXiv preprint arXiv:2403.08834 (March 13, 2024). https://doi.org/10.48550/arXiv.2403.08834.
Croskerry, Pat. “Cognitive Forcing Strategies in Clinical Decisionmaking.” Annals of Emergency Medicine 41, no. 1 (January 2003): 110–20. https://doi.org/10.1067/mem.2003.22.
Dankwa-Mullan, Irene. “Health Equity and Ethical Considerations in Using Artificial Intelligence in Public Health and Medicine.” Preventing Chronic Disease 21 (August 22, 2024): E64. https://doi.org/10.5888/pcd21.240245.
Enlitic. “Deidentifying and Anonymizing Healthcare Data.” Enlitic (blog), January 25, 2023. Accessed April 20, 2025. https://enlitic.com/blogs/deidentifying-and-anonymizing-healthcare-data/.
IBM. “What Is Explainable AI?” IBM Think, March 29, 2023. https://www.ibm.com/think/topics/explainable-ai.
Jensen, David, Brian LaMacchia, Ufuk Topcu, and Pamela Wisniewski. “Algorithmic Robustness.” arXiv preprint arXiv:2311.06275 (October 17, 2023). https://doi.org/10.48550/arXiv.2311.06275.
Johnson, Kevin B., Wei Qi Wei, Dilhan Weeraratne, Mark E. Frisse, Karl Misulis, Kyu Rhee, Juan Zhao, and Jane L. Snowdon. “Precision Medicine, AI, and the Future of Personalized Health Care.” Clinical and Translational Science 14, no. 1 (January 2021): 86–93. https://doi.org/10.1111/cts.12884.
Kaushal, Amit, Russ Altman, and Curt Langlotz. “Geographic Distribution of US Cohorts Used to Train Deep Learning Algorithms.” JAMA 324, no. 12 (2020): 1212–13. https://doi.org/10.1001/jama.2020.12067.
Kosinski, Matthew. “What Is Black Box AI and How Does It Work?” IBM Think, October 29, 2024. Accessed April 20, 2025. https://www.ibm.com/think/topics/black-box-ai.
Melvin Kranzberg, “Technology and History: ‘Kranzberg’s Laws,’” Technology and Culture 27, no. 3 (1986): 545.
Nardocci, Costanza. “Proxy Discrimination in Artificial Intelligence: What We Know and What We Should Be Concerned About.” Chaire de recherche du Canada sur la culture collaborative en droit et politiques de la santé blog, February 9, 2024. https://www.chairesante.ca/en/articles/2024/proxy-discrimination-in-artificial-intelligence-what-we-know-and-what-we-should-be-concerned-about/.
National Cancer Institute. “NIH Researchers Develop AI Tool with Potential to More Precisely Match Cancer Drugs to Patients.” Press release, April 18, 2024. https://www.cancer.gov/news-events/press-releases/2024/ai-tool-matches-cancer-drugs-to-patients.
Noyes, Jesse. “Perceptions of AI in Healthcare: What Professionals and the Public Think.” The Intake, Tebra. Last updated March 14, 2024. Accessed April 20, 2025. https://www.tebra.com/theintake/medical-deep-dives/tips-and-trends/research-perceptions-of-ai-in-healthcare.
Obermeyer, Ziad, and Sendhil Mullainathan. “Dissecting Racial Bias in an Algorithm That Guides Health Decisions for 70 Million People.” In Proceedings of the Conference on Fairness, Accountability, and Transparency, 89–101. New York, NY: Association for Computing Machinery, 2019. https://doi.org/10.1145/3287560.3287593.
Rezk, Eman, Mohamed Eltorki, and Wael El-Dakhakhni. “Improving Skin Color Diversity in Cancer Detection: Deep Learning Approach.” JMIR Dermatology 5, no. 3 (August 19, 2022): e39143. https://doi.org/10.2196/39143.
Ronen, Ophir. “Transparent AI In Healthcare: Transforming the Industry for the Better.” Forbes, December 5, 2023. Accessed April 20, 2025. https://www.forbes.com/councils/forbestechcouncil/2023/12/05/transparent-ai-in-healthcare-transforming-the-industry-for-the-better/.
Spatharou, Angela, Solveigh Hieronimus, and Jonathan Jenkins. Transforming Healthcare with AI: The Impact on the Workforce and Organizations. Executive Briefing. McKinsey & Company and EIT Health, March 10, 2020. https://www.mckinsey.com/industries/healthcare/our-insights/transforming-healthcare-with-ai.
Stanford HAI. “Should AI Models Be Explainable? That Depends.” Stanford HAI News, March 16, 2021. Accessed April 20, 2025. https://hai.stanford.edu/news/should-ai-models-be-explainable-depends.
Stanford Medicine News Center. “Artificial Intelligence Rivals Radiologists in Screening X‑Rays for Certain Diseases.” Stanford Medicine, November 15, 2017. http://med.stanford.edu/news/all-news/2018/11/ai-outperformed-radiologists-in-screening-x-rays-for-certain-diseases.html.
Wells, Claudia. “Inclusive Data to Leave No One Behind – Best Practices in Data Disaggregation and Use.” PDF file. UN Women, Commission on the Status of Women, 63rd Session, March 2019. https://www.unwomen.org/sites/default/files/Headquarters/Attachments/Sections/CSW/63/official-meetings/Claudia%20Wells%20updated.pdf.
Leave a Reply