AI/ML Devices: The Global Transparency Push

On June 13, the US FDA, Health Canada, and the UK’s Medicines and Health Care Products Regulatory Agency (MHRA) released a paper titled, “Transparency for Machine Learning-Enabled Medical Devices: Guiding Principles.” Here we break it down for Artificial Intelligence/ Machine Learning (AI/ML) companies and explain what it states, what it implies, and why this matters.

Manufacturers of AI/ML devices will need to prepare for intensified public scrutiny of their performance data. Small differences among subgroups, significant or not, could impact your marketing and sales messaging. You can minimize roadblocks to your marketing and sales by pursuing agreement on core claims and language during the review.

CONTEXT

These efforts at transparency are related to a shared central concern that regulatory bodies have with AI/ML technology:  

An algorithm/model malfunction is neither easily detected, nor easily corrected.

A common problem in getting an AI/ML device through review is addressing false positives and false negatives. It’s not that regulators expect perfection; it’s that it’s nearly impossible to determine a root cause and correct it. 

For traditional medical devices, the root cause for any failures or instances of subpar performance can usually be determined in several ways, such as physical inspection, reviews of error logs, or the examination of manufacturing processes and equipment.

In contrast, the “black box” nature of AI/ML means there’s no way to look under the hood and figure out exactly what went wrong. The manufacturer can train it more and hope it improves, but that’s an inexact solution. This raises the overall risk in the eyes of regulatory reviewers.

Consider this as well: can a user tell if your device has malfunctioned?

For many physical medical devices, they can, since there are alerts and warnings built in, or obvious signs like pumps not pumping, displays not turning on, or even unusual noises. 

For AI/ML devices, it may be difficult or impossible to tell, especially if the end user is a consumer rather than a medical doctor.

To address these issues, the regulators propose a set of guiding principles focused on transparency: 

“Effective transparency that helps parties make informed decisions can help control risks.”

“Transparency helps parties detect and investigate errors or a decline in performance.”

“...knowledge of how a device works and how it was developed can help identify bias and assess whether a system or output is justifiable.”

In other words, one of the main risk mitigations is educating the user so that they may make an informed decision on whether or not to trust the results. 

IMPACT ON MEDICAL DEVICE MANUFACTURERS

Highly detailed training and performance data will need to be provided not just to regulatory reviewers, but also to the device users.

In recent years we’ve watched as FDA reviewers have pressed for more and more detailed data analysis, not just for performance data but also for training data. There has also been a push to disclose more of this detailed data to users, rather than summary data as was previously the norm.

However, this is the first time that this has been put to paper as a concrete policy position. Negotiations are still possible, but companies will need to make a case for how the redaction of certain data does not increase risks to the users or patients. Any bias in the data or substandard performance for certain case types or demographics may lead to required disclaimers or data disclosures in marketing and sales materials.  

So, what can you do?

The guiding principles recommend the following types of information for disclosure: 

“[Information that] characterizes the training and testing data”

“... summaries of clinical studies”

“Known biases or failure modes”

“Confidence intervals associated with the provided outputs”

“Known gaps in the data characterization”

It may not be necessary to disclose all of this. So, as you go through review and negotiations with FDA, consider the following:

GUIDELINE 1

Ensure that there is agreement on what data will be included in the publicly available decision summary. Don’t be afraid to question why certain information needs to be public and don’t hesitate to propose alternative presentations or treatments of data.

GUIDELINE 2

Propose your marketing and sales language in your submission so that you can negotiate it during review. Be sure to get confirmation from FDA on what language and claims are and are not appropriate, as well as what disclaimers may be required. Keep your marketing team looped in on this so that they can get a head start crafting a compliant but effective strategy.

This includes anticipating what questions your customers might ask after seeing your data, and ensuring that you understand any limitations on how you can respond to those questions.

GUIDELINE 3

Consider when and where your data should be presented (user manual, website, sales decks, in the user interface, etc.). Review your product flows to identify how to do this without damaging the user experience and be ready to propose this during the review.

Diligence and preparation will pay off here, and you can avoid hobbling your marketing and sales strategy.   

A BOLD PREDICTION

There is another implication in this paper that may have an outsized impact, but that seems to have gone largely unnoticed: the joint recommendation for “human-centered design.”

This is more than just a buzzy term. It’s a direct reference to the international standard ISO 9241-210:2019, “Ergonomics of human-system interaction - Human-centred design for interactive systems.” 

This is a rarely-cited standard that describes best practice processes for design and product development throughout the entire product life cycle. The standard is a part of, and references, a large family of ISO standards that has been expanding in recent years to cover design and development recommendations for topics such as UI/ UX design, virtual and augmented reality, haptic systems, and other interface considerations. 

That so much work is currently being done to expand this family of standards, coupled with the fact that three prominent regulatory bodies have just publicly referenced it as the basis for compliance with the proposed policies, suggests that the ISO 9241 family is about to become a lot more important. 

Consider reviewing ISO 9241-210:2019, comparing your current product development procedures to the included checklist, and drafting updated product development procedures that align with this standard. Human-centered design principles may soon become an expectation rather than a recommendation.