Add us as a preferred source
Nik Kairinos, chief executive and co-founder of RAIDS AI, warns what needs to be done as AI in healthcare moves into frontline care.
Global healthcare provision has been under strain for years, and successive governments across the world are struggling with how to care for a growing and ageing population. According to the World Health Organization (WHO), by 2030, one in six people will be aged 60 or over.
This has led to a rising demand for care, but within an environment of workforce shortages and pressure on budgets. The WHO also reports a global shortage of 11 million healthcare workers by 2030, while the World Bank notes that healthcare spending has stagnated.
AI offers a way to increase capacity, improve diagnosis and make better use of healthcare data. The WHO has already highlighted applications spanning diagnosis, treatment, patient care, drug development and health-system management, for example.
However, the value that AI can bring to healthcare delivery depends on whether providers and patients can trust it once it is being used day-to-day.
AI is moving closer to the patient
The use of AI in healthcare has gradually increased over recent years, but there’s also been a notable shift as it moves from admin and support functions into frontline care, including clinical decision-making and treatment delivery.
Now, AI can be used to ease pressure on clinicians by analysing medical imaging, recommending and creating personalised treatment plans, monitoring patients and identifying patients at risk of deterioration. For example, the US FDA says AI-enabled medical devices are already being used for disease detection, diagnosis, personalised diagnostics and therapeutics, and clinical decision support.
This helps, as technology can speed up treatment and ensure that healthcare professionals can focus on the most complex cases. However, the more power and influence that AI has over patients’ health, the greater the consequences when its outputs are wrong.
Monitoring AI in healthcare: a new challenge
Traditional medical-device assurance has largely been built around establishing that a product is safe and effective before deployment, with subsequent modifications managed afterwards.
AI complicates that model because its performance can be affected by the environment in which it operates. The data reaching a model can change, patient populations can differ between hospitals or regions, clinical practice can evolve, and models can encounter inputs they were not exposed to during development. For example, the FDA explicitly warns that changes in data-acquisition systems, protocols and patient populations can lead to differences between an AI model’s development performance and its performance in clinical use.
When AI’s functions shift to care delivery, the implications of these changes and the effect on the output are potentially serious and far-reaching. They could manifest as falling diagnostic accuracy, demographic bias, increasing false positives or false negatives, unusual inputs or unexpected changes in recommendations, for example. For clinicians and their patients, the AI’s performance can directly affect a patient’s health.
Regulation is moving towards continuous assurance
Because of these direct consequences, healthcare already has stringent regulatory requirements, but regulators increasingly accept that pre-deployment testing alone is not enough for AI.
The direction of travel is already clear. Regulators and governing bodies increasingly recognise that AI needs to be monitored continuously, rather than regarded as safe simply because it passed assessment in testing. The WHO identifies evaluation and monitoring of AI-system performance as an important regulatory consideration for healthcare AI.
Further, Article 72 of the EU AI Act requires post-market monitoring of high-risk AI throughout its lifetime, while since the 2026 Digital Omnibus amendments, AI embedded in medical devices comes within the Act’s high-risk regime from 2 August 2028. Additionally, under Article 26(5) of the EU AI Act, hospitals deploying high-risk AI systems will be required to monitor their operation, identify emerging risks and report serious incidents. These obligations will apply from December 2027 for certain high-risk AI systems.
In Great Britain, strengthened MHRA post-market surveillance rules for medical devices have applied since 16 June 2025. The principle is that demonstrating safety and performance before a device reaches the market is not enough: manufacturers must continue to actively monitor how it performs once in use, collect and analyse post-market data, identify emerging risks and take corrective action where necessary.
Globally, ISO 42001 gives organisations a structured management-system framework for governing AI risks and demonstrating responsible AI management. It applies to organisations that develop, provide or use AI systems, making it relevant both to healthcare providers deploying AI and to the technology companies supplying it. ISO 42001 certification of an AI supplier can give healthcare organisations greater confidence that it has formal processes for governance, accountability and risk management.
Continuous monitoring
ISO 42001 moves AI governance beyond a point-in-time exercise and requires organisations to monitor, measure, analyse, and evaluate the performance of their AI management system, using the results to support continual improvement. Continuous, independent monitoring can support the continual improvement required by ISO 42001 by giving organisations ongoing visibility into how AI systems are performing and where risks or corrective action may be emerging.
Importantly, such monitoring provides an ongoing view of how AI behaves in production, rather than relying solely on testing carried out before deployment or point-in-time checks. It can monitor behaviour against a known baseline to help identify deviation from baseline behaviour, model or data drift, unexpected outputs, changes in performance, and emerging bias. Continuous monitoring also creates an audit trail, helping healthcare organisations investigate historically what happened if an AI-supported clinical decision is questioned.
Of course, monitoring does not replace clinical oversight, validation or regulation. But it provides another layer that can reveal risks that point-in-time assessment cannot and is an important tool for managing tools that might change and adapt after deployment.
The commercial argument
Healthcare organisations recognise the benefits AI can bring and are looking for reassurance that they can be confident in the AI they are using. The responsibility to instil that confidence increasingly falls on vendors selling the tools to make observability, monitoring and auditability part of the product architecture, rather than an afterthought.
In such a sector where the impact of mistakes can have significant consequences, vendors selling into healthcare should be able to answer questions such as: How will the customer know if model behaviour changes? What baseline is performance being compared against? Can anomalous behaviour be identified quickly?
This is a commercial argument as well as a regulatory one, because demonstrating that AI can be independently governed once deployed could increasingly become part of winning healthcare customers. Ultimately, AI vendors should not just sell healthcare organisations a model; they need to provide the means to assure its behaviour throughout its operational life too.
