Presenting an approach to synthesize quantitative and qualitative information from systematic reviews of multiple health interventions.

Within the context of an EUnetHTA multi-health technology assessment of twenty-three surgical techniques, we developed synthetic single tables, using color gradients and abbreviations, with information on which technologies had been compared, estimates of the size of differences for available comparisons, their clinical relevance, and certainty of the related evidence.

The proposed methodology provided, through a single depiction, information normally included in multiple figures/tables such as network plots, league tables, and summary of findings tables.

Transferring information on benefits, risks, and certainty of the available evidence on health interventions may be challenging, especially when assessing multiple treatments: more pieces of information need to be integrated in order to show an overall picture for each of the chosen outcomes, and usual reporting tools may be targeted to researchers more than to different kinds of decision makers. While more in-depth layers of information can always be added to satisfy needs of different audiences, the proposed tools could favor a quick interpretation of articulated scientific data by both decision makers and researchers.

Patient and public involvement (PPI) in the Brazilian Health Technology Assessment (HTA) process occurs in response to a legislative mandate for “social participation.” This resulted in some limited patient participation activities, and, therefore, a more systematic approach was needed. The study describes the development of a suggested framework for action to improve PPI in HTA.

This work used formal methodology to develop a PPI framework based on three-phase mixed-methods research with desktop review of Brazilian PPI activities in HTA; workshop, survey, and interviews with Brazilian stakeholders; and a rapid review of international practices to enact effective patient involvement. Patient partners reviewed the draft framework.

According to patient group representatives, their involvement in the Brazilian HTA process is important but could be improved. Different stakeholders perceived barriers, identified values, and made suggestions for improvement, such as expansion of communication, capacity building, and transparency, to support more meaningful patient involvement. The international practices identified opportunities for earlier, more active, and collaborative PPI during all HTA stages, based on values and principles that are relevant for Brazilian patients and the public. These findings were synthesized to design a framework that defines and systematizes actions to support PPI in Brazil, highlighting the importance of evaluating these strategies.

Since the publication of this framework, some of its suggestions are being implemented in the Brazilian HTA process to improve PPI. We encourage other HTA organizations to consider a systematic and planned approach with regular evaluation when pursuing or strengthening involvement practices.

Mini health technology assessment (HTA) reports have been used to support policy makers and health systems by providing a timely summary of scientific evidence. The objective of this meta-epidemiologic study was to evaluate the quality of reporting of mini-HTA reports published in Brazil.

An electronic search for all mini-HTA reports published between 2014 and March 2019 was conducted in the SISREBRATS and CONITEC databases. The study selection and data extraction were performed by two independent assessors. The following data were extracted: bibliographic data; research question; characteristics of the population, health technologies and outcomes assessed; eligibility criteria; information about searches and study selection; risk of bias assessment; quality of evidence assessment; synthesis of results; and recommendation about the technology evaluated. A descriptive analysis was used to summarize the information retrieved from all the included mini-HTA reports.

We included 103 mini-HTA reports, the great majority of which (92.3 percent) focused on the coverage of the technologies in the healthcare system, with more than 60 percent being about drugs. Only five mini-HTA reports (4.8 percent) gave reasons for the choice of outcomes, and fifteen (14.5 percent) discriminated between primary and secondary outcomes. All mini-HTAs reported the databases searched and 99 percent of them reported using Medline. Sixty percent of the mini-HTA reported assessing the risk of bias, and 52 percent reported assessing the quality of evidence.

The quality of reporting of the mini-HTA reports performed in Brazil is insufficient and needs to be improved to guarantee transparency and replicability.

In 2019, members of the Health Technology Assessment international (HTAi) Interest Group for Disinvestment and Early Awareness (DEA-IG) and the HTAi Interest Group for Information Retrieval (IR-IG) agreed to produce quarterly current awareness alerts for members of the DEA-IG. The purpose was to pilot a predefined strategy for sharing new publications on methods and topical issues in this area.

Literature search strategies for PubMed and Google were developed. Retrieved citations were posted on the DEA-IG Web site. Members of the DEA-IG received an email notification when new alerts were available. An informal survey of the DEA-IG members was used to provide feedback after the pilot.

Six alerts were issued during the pilot (June 2019–September 2020) with a total of 170 citations. The bulk of the information were 124 PubMed indexed citations, and of these, 96 were retrieved by the PubMed search strategies. Google searches were not found to be useful, but ongoing horizon scanning work at the Canadian Agency for Drugs and Technologies in Health (CADTH) provided additional information. Based on retrospective sorting, we considered thirty-five PubMed citations to be highly relevant for health technology assessment (HTA). The response rate to the survey was limited (seventeen respondents), but most respondents found the alerts useful for their work.

The results of this pilot project can be used to revise search strategies and information sources, improve the relevance of the alerts, and plan for expanded dissemination strategies.

We evaluated the early impact of a new hospital-based health technology assessment (HB-HTA) program, called Smart Innovation, at the University of Washington Medical Center (UWMC).

We compared the UWMC's utilization trends for two surgical procedures to control hospitals by evaluating the difference before and after adoption decisions: (i) a new filter for transcatheter aortic valve replacement (TAVR) procedures that treat aortic valve stenosis and (ii) microwave ablation (MWA) for treating hepatocellular carcinoma. We used descriptive statistics to assess the difference between the UWMC and controls for TAVR and MWA procedures and multivariate difference-in-differences (DID) analyses to test for statistical significance.

The UWMC experienced a 10 percent reduction in TAVR procedures compared with controls following the implementation of the TAVR Sentinel filter. The DID regression model indicated a 1.5 reduction in the number of TAVR procedures per quarter at the UWMC between the pre- and post period, which was not statistically significant (p-value: .87). The UWMC experienced a 51 percent reduction in utilization when compared with controls for MWA procedures in the pre- and post periods. The DID model for MWA indicated an 18.8 decrease in utilization per quarter during the study period for the UWMC, which was statistically significant (p-value: .0007). For MWA procedures, the UWMC experienced a $647,658 dollar reduction in total costs in the post period compared with controls.

When the UWMC used HB-HTA methods for technology adoption, there was a reduction in utilization and total costs when compared with controls; however, when the UWMC adopted a new technology without using HB-HTA methods, there was no difference in utilization.

Peer review of searches is a process whereby both the search strategies and the search process description are reviewed, ideally using an evidence-based checklist.

As the search strategy underpins any well-conducted evidence synthesis, its quality could affect the final result. Evidence shows, however, that search strategies are prone to error.

There is increasing awareness and use of the PRESS Evidence-Based Checklist and peer review of search strategies, at the outset of evidence syntheses, prior to the searches being run, and this is now recommended by a number of evidence synthesis organizations.

Searches for evidence syntheses should be peer reviewed by a suitably qualified and experienced librarian or information specialist after being designed, ideally, by another suitably qualified and experienced librarian or information specialist. Peer review of searches should take place at two important stages in the evidence synthesis process; at the outset of the project prior to the searches being run and at the prepublication stage. There is little empirical evidence, however, to support the effectiveness of peer review of searches. Further research is required to assess this. Those wishing to stay up to date with the latest developments in information retrieval, including peer review of searches, should consult the SuRe Info resource (http://www.sure-info.org), which seeks to help information specialists and others by providing easy access to the findings from current information retrieval methods research and thus support more research-based information retrieval practice.

The European Network for Health Technology Assessment (EUnetHTA) was established in 2006 and comprises over eighty organizations from thirty European countries. In its fifth project phase (Joint Action 3), EUnetHTA set up a quality management system (QMS) to improve the efficiency and standardization of joint work. This article presents EUnetHTA's new QMS and outlines experiences and challenges during its implementation.

Several working groups defined processes and methods to support assessment teams in creating high-quality assessment reports. Existing guidelines, templates, and tools were refined and missing parts were newly created and integrated into the new QMS framework. EUnetHTA has contributed to Health Technology Assessment (HTA) capacity building through training and knowledge sharing. Continuous evaluation helped to identify gaps and shortcomings in processes and structures.

Based on a common quality management concept and defined development and revision procedures, twenty-seven partner organizations jointly developed and maintained around forty standard operating procedures and other components of the QMS. All outputs were incorporated into a web-based platform, the EUnetHTA Companion Guide, which was launched in May 2018. Concerted efforts of working groups were required to ensure consistency and avoid duplication.

With the establishment of a QMS for jointly produced assessment reports, EUnetHTA has taken a significant step toward a sustainable model for scientific and technical collaboration within European HTA. However, the definition of processes and methods meeting the numerous requirements of healthcare systems across Europe remains an ongoing and challenging task.

To describe the type and level of ethical integration in published health technology assessment (HTA) reports and systematically identify the ethical approaches utilized.

A literature search was conducted with the Google™ search engine using the keyword “ethic” between 1 January 2015 and 20 August 2019. Only HTA assessment reports with a section on ethics were retained and classified according to their level of ethical integration: no ethical analysis, ethical issues highlighted, assessments according to legal or social norms, and assessments from a moral or axiological perspective—using a qualitative methodology to distinguish such integration.

This review yielded 188 reports with a section identified as being on ethics, produced by seventeen HTA agencies in eleven countries. One hundred and thirty-six reports did not develop an ethical analysis, thirty-one highlighted ethical issues, seventeen conducted a norm-based ethical assessment using a descriptive approach grounded in social norms, and four developed an assessment grounded in a moral or axiological perspective. The bioethical “four-principles” framework was used, but mainly for presenting ethical issues and not as a moral framework.

The majority of reports featuring a section on ethics mention ethical considerations without ethical analysis. Ethical issues are grouped with legal, social, and organizational issues and treated as contextual considerations that decision makers should be aware of. When reports present systematic norm-based ethical assessments from a descriptive perspective or ethical assessment based on a moral or axiological perspective, there is a tendency to ground these analyses in frameworks created for the purpose and reliant on a concept of ethics supporting them.

This article aims to provide a brief review of information retrieval and hospital-based health technology assessment (HB-HTA) and describe library experiences and working methods at a regional HB-HTA center from the center's inception to the present day.

For this brief literature review, searches in PubMed and LISTA were conducted to identify studies reporting on HB-HTA and information retrieval. The description of the library's involvement in the HTA center and its working methods is based on the authors’ experience and internal and/or unpublished documents.

Region Västra Götaland is the second largest healthcare region in Sweden and has had a regional HB-HTA center since 2007 (HTA-centrum). Assessments are performed by clinicians supported by HTA methodologists. The medical library at Sahlgrenska University Hospital works closely with HTA-centrum, with one HTA librarian responsible for coordinating the work.

In the literature on HB-HTA, we found limited descriptions of the role librarians and information specialists play in different units. The librarians at HTA-centrum play an important role, not only in literature searching but also in abstract and full-text screening.

Methodologies of health technology assessment (HTA) for medical technologies are well established; yet, operational frameworks that enable appropriate uptake of HTA recommendations into routine clinical practice are lacking. This review aims to identify the key themes needed to guide the planning and implementation of HTA subsidy decisions for medical technologies such as diagnostics, medical devices, and services and to monitor their impact on a complex multipayer healthcare system like Singapore.

A literature search of implementation frameworks was conducted up to 20 December 2020 and was documented in a flow diagram. A thematic analysis of the evidence base was performed using the Braun and Clark approach to identify key themes, from which an implementation framework suitable for Singapore's healthcare system could be developed.

The searches yielded forty-four articles for review, from which twenty themes were identified. The top ten themes constituted the key themes of implementation essential for local adaptation and were categorized into five domains: implementation strategy, organizational support, stakeholder engagement, information dissemination, and implementation outcomes and evaluation. These domains provide operational guidance to methodically identify gaps to facilitate sustainable implementation of HTA-informed medical technology subsidy decisions.

A robust and adaptable implementation of HTA-informed subsidy decisions is crucial to optimize its intended impact of improving patient outcomes per dollar spent. The key themes of an implementation framework should capture the important aspects of organizational feasibility to ensure successful adoption in a complex multipayer healthcare system like Singapore.

Health technology assessment (HTA) can impact health inequities by informing healthcare priority-setting decisions. This paper presents a novel checklist to guide HTA practitioners looking to include equity considerations in their work: the equity checklist for HTA (ECHTA). The list is pragmatically organized according to the generic HTA phases and can be consulted at each step.

A first set of items was based on the framework for equity in HTA developed by Culyer and Bombard. After rewording and reorganizing according to five HTA phases, they were complemented by elements emerging from a literature search. Consultations with method experts, decision makers, and stakeholders further refined the items. Further feedback was sought during a presentation of the tool at an international HTA conference. Lastly, the checklist was piloted through all five stages of an HTA.

ECHTA proposes elements to be considered at each one of the five HTA phases: Scoping, Evaluation, Recommendations and Conclusions, Knowledge Translation and Implementation, and Reassessment. More than a simple checklist, the tool provides details and examples that guide the evaluators through an analysis in each phase. A pilot test is also presented, which demonstrates the ECHTA's usability and added value.

ECHTA provides guidance for HTA evaluators wishing to ensure that their conclusions do not contribute to inequalities in health. Several points to build upon the current checklist will be addressed by a working group of experts, and further feedback is welcome from evaluators who have used the tool.

Solutions like crowd screening and machine learning can assist systematic reviewers with heavy screening burdens but require training sets containing a mix of eligible and ineligible studies. This study explores using PubMed's Best Match algorithm to create small training sets containing at least five relevant studies.

Six systematic reviews were examined retrospectively. MEDLINE searches were converted and run in PubMed. The ranking of included studies was studied under both Best Match and Most Recent sort conditions.

Retrieval sizes for the systematic reviews ranged from 151 to 5,406 records and the numbers of relevant records ranged from 8 to 763. The median ranking of relevant records was higher in Best Match for all six reviews, when compared with Most Recent sort. Best Match placed a total of thirty relevant records in the first fifty, at least one for each systematic review. Most Recent sorting placed only ten relevant records in the first fifty. Best Match sorting outperformed Most Recent in all cases and placed five or more relevant records in the first fifty in three of six cases.

Using a predetermined set size such as fifty may not provide enough true positives for an effective systematic review training set. However, screening PubMed records ranked by Best Match and continuing until the desired number of true positives are identified is efficient and effective.

The Best Match sort in PubMed improves the ranking and increases the proportion of relevant records in the first fifty records relative to sorting by recency.

The Patient-Centered Outcomes Research Institute (PCORI) horizon scanning system is an early warning system for healthcare interventions in development that could disrupt standard care. We report preliminary findings from the patient engagement process.

The system involves broadly scanning many resources to identify and monitor interventions up to 3 years before anticipated entry into U.S. health care. Topic profiles are written on included interventions with late-phase trial data and circulated with a structured review form for stakeholder comment to determine disruption potential. Stakeholders include patients and caregivers recruited from credible community sources. They view an orientation video, comment on topic profiles, and take a survey about their experience.

As of March 2020, 312 monitored topics (some of which were archived) were derived from 3,500 information leads; 121 met the criteria for topic profile development and stakeholder comment. We invited fifty-four patients and caregivers to participate; thirty-nine reviewed at least one report. Their perspectives informed analyst nominations for fourteen topics in two 2019 High Potential Disruption Reports. Thirty-four patient stakeholders completed the user-experience survey. Most agreed (68 percent) or somewhat agreed (26 percent) that they were confident they could provide useful comments. Ninety-four percent would recommend others to participate.

The system has successfully engaged patients and caregivers, who contributed unique and important perspectives that informed the selection of topics deemed to have high potential to disrupt clinical care. Most participants would recommend others to participate in this process. More research is needed to inform optimal patient and caregiver stakeholder recruitment and engagement methods and reduce barriers to participation.

This paper aims to describe the development of a flowchart to guide the decisions of researchers in the Spanish Network for Health Technology Assessment of the National Health System (RedETS) regarding patient involvement (PI) in Health Technology Assessment (HTA). By doing so, it reflects on current methodological challenges in PI in the HTA field: how best to combine PI methods and what is the role of patient-based evidence.

A decisional flowchart for PI in HTA was developed between March and April 2019 following an iterative process, reviewed by the members of the PI Interest Group and other RedETS members and validated during an online deliberative meeting. The development of the flowchart was based on a previous methodological framework assessed in a pilot study.

The guidelines on how to involve patients in HTA in the RedETS were graphically represented in a flowchart. PI must be included in all HTA reports, except those that assess technologies with no relevant impact on patients’ experiences, values, and preferences. Patient organizations or expert patients related to the topic of the HTA report must be identified and invited. These patients can participate in protocol development, outcomes' identification, assessment process, and report review. When the technology assessed affects in a relevant way patient experiences, values, and preferences, patient-based evidence should be included through a systematic literature review or a primary study.

The decisional flowchart for PI in HTA contributes to the current methodological challenges by proposing a combination of direct involvement and patient-based evidence.

The frameworks used by Health Technology Assessment (HTA) agencies for value assessment of medicines aim to optimize healthcare resource allocation. However, they may not be effective at capturing the value of antimicrobial drugs.

To analyze stakeholder perceptions regarding how antimicrobials are assessed for value for reimbursement purposes and how the Australian HTA framework accommodates the unique attributes of antimicrobials in cost-effectiveness evaluation.

Eighteen individuals representing the pharmaceutical industry or policy-makers were interviewed. Interviews were transcribed verbatim, coded, and thematically analyzed.

Key emergent themes were that reimbursement decision-making should consider the antibiotic spectrum when assessing value, risk of shortages, the impact of procurement processes on low-priced comparators, and the need for methodological transparency when antimicrobials are incorporated into the economic evaluation of other treatments.

Participants agreed that the current HTA framework for antimicrobial value assessment is inadequate to properly inform funding decisions, as the contemporary definition of cost-effectiveness fails to explicitly incorporate the risk of future resistance. Policy-makers were uncertain about how to incorporate future resistance into economic evaluations without a systematic method to capture costs avoided due to good stewardship. Lacking financial reward for the benefits of narrower-spectrum antimicrobials, companies will likely focus on developing broad-spectrum agents with wider potential use. The perceived risks of shortages have influenced the funding of generic antimicrobials in Australia, with policy-makers suggesting a willingness to pay more for assured supply. Although antibiotics often underpin the effectiveness of other medicines, it is unclear how this is incorporated into economic models.

Integration of ethics into technology assessment in healthcare (HTA) reports is directly linked to the need of decision makers to provide rational grounds justifying their social choices. In a decision-making paradigm, facts and values are intertwined and the social role of HTA reports is to provide relevant information to decision makers. Since 2003, numerous surveys and discussions have addressed different aspects of the integration of ethics into HTA. This study aims to clarify how HTA professionals consider the integration of ethics into HTA, so an international survey was conducted in 2018 and the results are reported here.

A survey comprising twenty-two questions was designed and carried out from April 2018 to July 2018. Three hundred and twenty-eight HTA agencies from seventy-five countries were invited to participate in this survey.

Eighty-nine participants completed the survey, representing a participation rate of twenty-seven percent. As to how HTA reports should fulfill their social role, over 84 percent of respondents agreed upon the necessity to address this role for decision makers, patients, and citizens. At a lower level, the same was found regarding the necessity to make value-judgments explicit in different report sections, including ethical analysis. This contrasts with the response-variability obtained on the status of ethical analysis with the exception of the expertise required. Variability in stakeholder-participation usefulness was also observed.

This study reveals the importance of a three-phase approach, including assessment, contextual data, and recommendations, and highlights the necessity to make explicit value-judgments and have a systematic ethical analysis in order to fulfill HTA's social role in guiding decision makers.

Health apps are software programs that are designed to prevent, diagnose, monitor, or manage conditions. Inconsistent terminology for apps is used in research literature and bibliographic database subject headings. It can therefore be challenging to retrieve evidence about them in literature searches. Information specialists at the United Kingdom's National Institute for Health and Care Excellence (NICE) have developed novel validated search filters to retrieve evidence about apps from MEDLINE and Embase (Ovid).

A selection of medical informatics journals was hand searched to identify a “gold standard” (GS) set of references about apps. The GS set was divided into a development and validation set. The filters’ search terms were derived from and tested against the development set. An external development set containing app references from published NICE products was also used to inform the development of the filters. The filters were then validated using the validation set. Target recall was >90 percent. The filters’ overall recall, specificity, and precision were calculated using all the references identified from the hand search.

Both filters achieved 98.6 percent recall against their validation sets. Overall, the MEDLINE filter had 98.8 percent recall, 71.3 percent specificity, and 22.6 percent precision. The Embase filter had 98.6 percent recall, 74.9 percent specificity, and 24.5 percent precision.

The NICE health apps search filters retrieve evidence about apps from MEDLINE and Embase with high recall. They can be applied to literature searches to retrieve evidence about the interventions by information professionals, researchers, and clinicians.

The Spanish Network of Agencies for Assessing National Health System Technologies and Performance (RedETS) defined a patient involvement (PI) framework for health technology assessment (HTA) activities in 2016. The aim of this study is to evaluate the process and impact of those PI initiatives that were implemented in the first year following the publication of this new framework.

A survey was sent to those HTA researchers who implemented PI in RedETS projects. Responses were reviewed by two authors. An adapted thematic analysis was performed and the results were later discussed by all authors.

Six responses from six agencies/units were analyzed. The objectives of PI initiatives were the following: inclusion of patient perspectives, preferences and values; elicitation of important health outcomes measures; and barriers, facilitators, or suggestions for implementation. Different methods were used for PI: surveys, focus groups, in depth interviews, and participation in an expert panel. Five main themes emerged: (i) challenges with the recruitment process, (ii) needs identified, (iii) impact of PI, (iv) lessons learned, and (v) suggestions for the future.

PI initiatives within the RedETS framework were tailored to each HTA project, its specific goals and the individual needs and resources of each HTA agency. The results also pointed out how PI has a relevant impact that has enriched RedETS products providing key information on experiences, values, and preferences of patients, contributions that benefit the HTA and the process of drawing up recommendations. The main challenges were related to recruitment processes and capacity building.

The objectives of the study were to establish a benchmarking tool to collect metrics to enable increased clarity regarding the differences and similarities across health technology assessment (HTA) agencies, to assess performance within and across HTA agencies, identify areas in the HTA processes in which time is spent and to enable ongoing performance improvement.

Common steps and milestones in the HTA process were identified for meaningful benchmarking among agencies. A benchmarking tool consisting of eighty-six questions providing information on HTA agency organizational aspects and information on individual new medicine review timelines and outcomes was developed with the input of HTA agencies and validated in a pilot study. Data on 109 HTA reviews from five HTA agencies were analyzed to demonstrate the utility of this tool.

This study developed an HTA benchmarking methodology, comparative metrics showed considerable differences among the median timelines from assessment and appraisal to final HTA recommendation for the five agencies included in this analysis; these results were interpreted in conjunction with agency characteristics.

It is feasible to find consensus among HTA agencies regarding the common milestones of the review process to map jurisdiction-specific processes against agreed metrics. Data on characteristics of agencies such as their scope and remit enabled results to be interpreted in the appropriate local context. This benchmarking tool has promising potential utility to improve the transparency of the review process and to facilitate both quality assurance and performance improvement in HTA agencies.

To investigate the behavior of restricted mean survival time (RMST) and designs of a two-state Markov microsimulation model through a 2 × 4 × 2 full factorial experiment.

By projecting patient-wise 15-year-post-trial survival, we estimated life-year-gained between an intervention and a control group using data from the Cardiovascular Outcomes for People Using Anticoagulation Strategies Study (COMPASS). Projections considered either in-trial events or post-trial medications. They were compared based on three factors: (i) choice of probability of death, (ii) lengths of cycle, and (iii) usage of half-a-cycle age correction. Three-way analysis of variance and post-hoc Tukey's Honest Significant Difference test compared means among factors.

When both in-trial events and post-trial study medications were considered, monthly, quarterly, or semiannually were not different from one other in projected life-year-gained. However, the annual one was different from the others: mean and 95 percent confidence interval 252.2 (190.5–313.9) days monthly, 251.8 (192.0–311.6) quarterly, 249.1 (189.7–308.5) semiannually, and 240.8 (178.5–303.1) annually. The other two factors also impacted life-year-gained: background probability (269.1 [260.3–277.9] days projected with REACH-based-probabilities, 227.7 [212.6–242.8] with a USA life table); half-a-cycle age correction (245.5 [199.0–292] with correction and 251.4 [209.1–293.7] without correction). When not considering post-trial medications, only the choice of probability of death appeared to impact life-year-gained.

For a large trial or cohort, to optimally project life-year-gained, one should consider using (i) annual projections, (ii) life table probabilities, (iii) in-trial events, and (iv) post-trial medication use.