Q&A with Health Technology Assessment expert Dr. Val Rac.
The value of AI-powered technologies in providing precision tools and insights for both patients and healthcare providers has become widely recognized. As a result, these technologies are becoming more prevalent in heart failure care. However, questions are being raised about the social, economic, and ethical impacts of these non-traditional tools. How is AI-powered technology being evaluated?
There are few better equipped to answer this question than Dr. Val Rac, Scientist and Health Technology Assessment Lead at the Peter Munk Cardiac Centre and Director of the Program for Health System and Technology Evaluation. Dr. Rac and her team of international collaborators received a TRANSFORM HF Collaboration Starter Grant in 2021 for their work in developing a Health Technology Assessment framework specific to AI-powered technologies. Her team’s work will be culminating at the end of May (May 24th) with a validation workshop to test this framework. We sat down with Dr. Rac to explore this emerging field and learn more about her team’s preliminary findings.
Q: What is Health Technology Assessment?
Health Technology Assessment (HTA) involves the complex analysis of new, promising technology and the value it may bring to the healthcare system and society overall. It is often used to inform policy decisions.
Traditional HTA is conducted for devices that have already received regulatory approval and are present on the market but must be evaluated in terms of benefits and costs. This evaluation is informed by “domains” such as clinical evidence; safety; effectiveness; patient preferences; ethical, legal, social, and economic impacts; and feasibility of implementation. Feasibility can refer to operational feasibility (how will the technology be embedded into existing systems?) and economic feasibility (how much will the technology cost the system?). Feasibility is a very important consideration, as it reflects the ability of the technology in question to be sustainably implemented to bring continuous values to patients, providers, and systems.
Early HTA is used to evaluate technology that has not yet reached the market. In this case, evaluation is focused on describing the potential disruptiveness of the new technology and investigating the healthcare system’s, and society’s, readiness and willingness to pay for the technology. These findings can help tailor technology modifications to suit better identified thresholds or preferences.
Q: AI-powered technology is becoming more prevalent. How is HTA adapting to assess such technology? What are the main challenges?
When we began our project, there was no specific HTA framework for AI-powered technologies. Our group has since developed and begun adapting and validating a new framework. During the early stages of this process, we found that many of the domains previously emphasized in traditional HTA frameworks were still relevant. However, the emphasis of these domains is where the primary differences occur.
AI-powered technology poses new ethical and legal challenges surrounding the type and quality of data being used and the way it is shared. For example, unlike a traditional blood pressure monitoring device that uses data from the patient using it, AI-powered technologies will need to be trained on large data sets from different patients or populations. This brings up questions of ethics and equity.
Essentially, deeper and more thoughtful exploration is warranted for some domains in AI-powered technology HTA compared to more traditional HTA.
Q: You’re part of an international collaboration to validate a HTA AI model. Can you tell us more about this collaboration and your findings to date?
This project is being conducted through an international collaboration between the University Health Network and the Odense University Hospital in Demark. The collaboration was formed well ahead of this project’s inception, and the TRANSFORM HF Collaboration Starter Grant has allowed the group’s previous work to be expanded upon.
The first three phases of this project were conducted in Denmark. First, the Danish team undertook a scoping review on existing HTA for AI-powered technologies, which provided insight into the best domains to include in a framework. Next, the Danish team interviewed six leading researchers in AI in Denmark. Further, the Danish team held two rounds of consultations with decision-makers, patient organizations, and researchers to develop a draft HTA framework, which we call the Model for Assessment of Artificial Intelligence (MAS-AI) framework.
The final phase of the project will involve validating the draft MAS-AI framework in a Canadian setting. The TRANSFORM HF grant supports this phase.
An article outlining the first three phases of the project was published in the SSRN e-library in February of this year. The author group behind the MAS-AI article consists of: Iben Fasterholdt (PhD), Tue Kjølhede (PhD), M.D Mohammad Naghavi-Behzad (PhD Candidate), Thomas Schmidt (Associate Professor), Quinnie Rautalammi (Consultant), M.D. Malene Grubbe Hildebrandt (Associate Professor), Anne Gerdes (Professor), Astrid Barkler (Patient Representative), Kristian Kidholm (Professor), Valeria E. Rac (Associate Professor), and M.D. Benjamin S. B. Rasmussen (Associate Professor).
Q: Tell me more about the MAS-AI Framework, what sets it apart from other assessment models?
Well, to begin, it was developed through a very rigorous, scientific, stepwise approach – from the scoping review, to the consultations in Denmark, to validation in Canadian settings.
But what really sets it apart is that it’s a framework explicitly developed for AI-powered technologies. In fact, in addition to a few viewpoint papers, this will be the first HTA framework for AI-powered technologies in Canada, and also internationally.
I am hoping that this framework will allow us to assess the value of different AI-powered heart failure technologies to ensure relevance, safety, and technical and cost-effectiveness, among other attributes. It is our goal to improve patient care and provide value for the healthcare system without compromising ethical or privacy principles.
Q: Can you walk us through the Canadian MAS-AI Validation Workshop and what you hope to accomplish?
Essentially, we are hoping to validate the MASAI framework in a Canadian context. Through the workshop, we will test the robust framework drafted from the Danish researchers, with considerations to emphasizing or expanding certain domains, as well as implementation challenges.
We will gather Canadian HTA and AI scientists, policy makers, patient partners, and institutional decision makers and present them with the current findings of the project, including the draft framework. We will then split them into working groups to discuss crucial topics and domains when evaluating AI technologies using the framework.
Afterwards, we will ask them to participate in a Delphi survey to assess the importance of different domains, and make any suggestions specific to the Canadian context.
As the framework was initially developed in a Danish context with a public health care system, we are expecting to make some modifications to the domains and framework as a result of this workshop. For example, Canada has a more culturally heterogenous population than Denmark, which must be considered when assessing AI-powered technologies. Also, we are cognizant that privacy laws might differ between Canada and Denmark.
Q: What are the next steps of this project?
Nationally, the next steps for this project are to pilot the framework with more technologies and involve policy makers more comprehensively so that they begin to use the framework, or at least specific domains, in their evaluative approaches. Ideally, we’d like the framework to become widely accepted.
Internationally, we also dream big! We want to evaluate the framework in different settings – possibly the UK, the Netherlands, and Australia. This collaboration started in the international context, so we want to expand that.
Q: How has TRANSFORM HF support impacted this undertaking?
TRANSFORM HF’s support was so much needed – it was truly our seed funding. It was fantastic to bring people together, especially in our case of an international collaboration.
You may have good will, but without funding it’s difficult to turn your good will into something concrete. The grant funding allowed us to start the conversation with a different flavour: “We have funding, we can set up the workshop.” Of course, then we attracted more funding.
For us, TRANSFORM HF’s support was the first big, concrete step beyond good will and common research interests.
MAS-AI was developed by a multidisciplinary group, including HTA experts; health economists; clinicians; technical, legal, and ethical experts; and patients.
HTA expert Dr. Iben Fasterholdt
Professor Kristian Kidholm
Professor Anne Gerdes
Medical doctor Dr. Benjamin Rasmussen
