The issue with the underlying healthcare data in AI

Bias in UK healthcare data is a practical information management concern that carries significant legal and compliance risks for organisations using AI.

AI is the word of the moment. Some, as I was, are sceptics. It’s just a bubble, it’s only pattern matching, it’s not ‘smart’ and cannot be relied upon. Others believe it will replace humans, whether by taking our jobs or something a little more I-Robot, perhaps even bringing about the end of the world.

No plan, no AI

Without a crystal ball, none of us truly knows where this AI journey will take us, or whether it will bring fundamental positive change to how we work and interact. Two things are certain: it is a helpful tool, and if we have the want for it to be a net positive, we cannot mindlessly follow the trend.

Innovation breeds opportunity, particularly in healthcare and especially from a patient’s perspective (and perhaps from the personal perspective of healthcare professionals). NHS waiting times are at an all-time high, important life-changing diagnoses take longer than they should, and healthcare professionals are being underpaid and overworked. More efficient ways of working, whether booking systems, diagnostic tools or treatment dispensers, could help eradicate time-consuming processes older than most of the people using the service.

Data-first planning

That can only be an improvement, right?

In theory, yes. But the issue I foresee is that AI output is only as good as its training data. What is the quality of that data?

Another, more long-term concern is what overreliance on AI could do to the quality of healthcare professionals. If critical thinking is like a muscle, what atrophy will they suffer?

For now, the more immediate and serious issue is the dataset underpinning AI in healthcare.

Bias in healthcare data

Why focus on data before issues, such as resistance to innovation or the NHS’s technical debt? As a woman, I have personally experienced bias and difficulties in diagnosis within the healthcare system.

This anecdotal experience is supported by recent studies. Women are diagnosed seven to ten years later than men, frequently after developing complications (Delanerolle et al., “Mind the Women’s Health Data Gap,” Biomedical Journal of Scientific & Technical Research, 2025). Despite women representing 51% of the population, we are drastically underrepresented in healthcare data.

Diagnosis and treatment inequality

Cardiovascular issues are a prime example. Women are less likely to receive guideline-recommended care, such as timely reperfusion therapy for myocardial infarction, coronary angiography, cardiac rehabilitation referral, and statin prescription. These gaps increase 30-day mortality risk after a heart attack (Wilkinson et al., Heart, 2019).

The ‘male default’ in heart disease diagnosis persisted until 1999, when the American Heart Association published its Guide to Preventive Cardiology for Women, recognising women’s different symptom presentations (AAMC, 2024).

Clinical trial inequality

Likewise, in clinical trials, an analysis of 1,433 trials (2016–2019) with over 300,000 participants found that, on average, only 41.2% of participants were female (Published in Contemporary Clinical Trials, 2022) and that as few as 22% of Phase I trial patients are female (Medidata, 2026, citing SciencePharma).

If the underlying datasets for diagnostic tools or drug testing are not improved to include female physiology, the biases and missed diagnoses we already see will only be compounded with the implementation of AI. In broad terms, the AI’s outputs will reflect the data they are trained on.

Early examples of biased AI output in HealthTech

This concern is already starting to emerge in the early adoption of AI. A University College London study found that AI models designed to predict liver disease from blood tests were twice as likely to miss diagnoses in women, largely because they were trained predominantly on male date(Pharma’s Almanac, 2025, citing Straw & Wu, BMJ Health Care Informatics, 2022).

Reasons and remedies

Historically, one justification for limited female participation in research has been that women’s bodies are too complicated, with too many variables to consider. I’m not a healthcare professional, but I can see some logic in this argument. From my limited knowledge of female hormones, I understand that where a woman is in her cycle could impact or change the results of a study for her as an individual.

But isn’t that even more reason to figure out this issue? Having this data available before a pharmaceutical product is put to market could prevent women being put in the position of only discovering adverse reactions after taking the medicine or undergoing a procedure, and after suffering harm.

Similar issues are known to arise for ethnic minority groups.

Perhaps the use of AI in formulating testing that accounts for these additional factors is the good early use case.

As far as I can see, AI is a helpful tool that can assist us in figuring out complex issues that we might not otherwise have the time or opportunity to sort out. With recent measures like the Renewed Women’s Health Strategy for England, published on 15 April 2026, which included a £1.5 million FemTech fund, could this be the opportunity to prioritise improving the underlying data before we jump headfirst into a messy AI bubble?

Existing system biases

Without an overhaul of the basis and categorisation of the data, AI is going to exacerbate existing biases in the UK healthcare system. A prime example is gynaecological waiting times, which are the longest in the country. This includes women waiting for surgery for endometriosis, Polyendocrine Metabolic Ovarian Syndrome (PMOS) and other serious pelvic pain conditions. Such surgery is categorised within the NHS system as “benign gynaecology” meaning that when weighed against other procedures, booking managers who are unaware of the clinical reality may deprioritise it, not understanding the seriousness of these operations for some women.

This creates human bias in the booking process and a lack of prioritisation, even when humans are controlling the system. That problem will only worsen without human intervention if an AI, lacking real-world context, is left to filter these decisions alone.

Practical recommendations

It is reassuring that steps have already been taken to consider these issues. I recently attended a webinar hosted by the Incubator for AI and Digital Healthcare, featuring Dr MaryAnn Ferreux, Chief Medical Officer at Health Innovation Kent Surrey Sussex, who has conducted extensive research on this topic. She made some practical recommendations, including:

• Increasing awareness of gender biases in AI
• Improving users’ understanding of AI itself (how it works, what it’s trained on, and how to use it without overreliance)
• Increasing the number of women in AI leadership, especially in health AI development
• Ensuring better procurement and governance within healthcare AI, implementing best practice policies, audit systems, bias detection and EDI requirements, to name a few.

Without such measures, AI risks misdiagnosing certain populations, overlooking sex-specific disease patterns, and widening existing health inequalities.

AI offers genuine promise for healthcare, which is needed now perhaps more than ever, but only if we are willing to confront the gaps in the data on which it is built. The opportunity is here; the question is whether we will seize it before history repeats itself in algorithmic form.