Search Filters to Identify Automation in HEOR: An Umbrella Review of Performance and Overlap
Downloads
DOI:
https://doi.org/10.71079/ASIDE.HS.122125202Abstract
Background: Search strategies used to identify evidence on automation in Health Economics and Outcomes Research (HEOR) often lack sensitivity and specificity, resulting in information overload or missed studies. This umbrella review evaluated and compared the performance and overlap of search filters commonly used to retrieve automation-related HEOR evidence.
Methods: Systematic literature reviews (SLRs) and search filters focusing on any form of automation in HEOR were included. Searches (January 01, 2023–July 03, 2024) were conducted in EMBASE, ISSG Resource, and Google Scholar. Subject headings, search terms, and performance metrics were extracted. Reference lists were cross-checked. Screening was performed by one reviewer, with 20% verified by a second reviewer. The PRESS checklist was used to assess search strategy quality. The protocol was registered with the Open Science Framework (OSF).
Results: Seven SLRs and one standalone filter, reporting 11 search strategies, met the inclusion criteria. HEOR relevance was defined by studies applying search filters in contexts of SLRs, indirect treatment comparisons, and economic modeling. Included SLRs retrieved between 5-273 studies. PubMed was the most frequently searched database. Commonly used subject headings included “artificial intelligence,” “deep learning,” “machine learning,” and “natural language processing,” with “artificial intelligence” the most frequent free-text term. Inclusion rates varied: title/abstract (1%–8%), full-text (27%–86%), and final inclusion (0.13%–2.31%). Time to include one study ranged from 0.6-8 hours.
Conclusions: Considerable variability in search filter performance was observed, causing lower specificity and inefficient evidence retrieval. Standardized, high-performing search strategies are needed to enhance efficiency and reliability in identifying automation-related HEOR evidence.
Keywords:
Automation, Artificial intelligence, Health economics and outcomes research, HEOR, Search filtersReferences
1. van Nooten F, Holmstrom S, Green J, Wiklund I, Odeyemi IA, Wilcox TK. Health economics and outcomes research within drug development: challenges and opportunities for reimbursement and market access within biopharma research. Drug Discov Today. 2012: 615 [PMID: 22366662, https://doi.org/10.1016/j.drudis.2012.01.021] DOI: https://doi.org/10.1016/j.drudis.2012.01.021
2. Reason T, Rawlinson W, Langham J, Gimblett A, Malcolm B, Klijn S. Artificial Intelligence to Automate Health Economic Modelling: A Case Study to Evaluate the Potential Application of Large Language Models. Pharmacoecon Open. 2024: 191 [PMID: 38340276, https://doi.org/10.1007/s41669-024-00477-8] DOI: https://doi.org/10.1007/s41669-024-00477-8
3. Nice. Use of AI in evidence generation: NICE position statement. 2024.
4. Mason J, Morrison A, Visintini S. An overview of clinical applications of artificial intelligence. Ottawa: CADTH. 2018:
5. Health NIoEi, Services S. Development and evaluation of a software tool based on GPT-4 to assist in sorting documents in the context of literature reviews: Literature, a proof of concept. 2024.
6. Escobar Liquitay CM, Garegnani L, Garrote V, Sola I, Franco JV. Search strategies (filters) to identify systematic reviews in MEDLINE and Embase. Cochrane Database Syst Rev. 2023: MR000054 [PMID: 37681507, https://doi.org/10.1002/14651858.MR000054.pub2] DOI: https://doi.org/10.1002/14651858.MR000054.pub2
7. Page MJ, McKenzie JE, Bossuyt PM, Boutron I, Hoffmann TC, Mulrow CD, Shamseer L, Tetzlaff JM, Akl EA, Brennan SE, Chou R, Glanville J, Grimshaw JM, Hrobjartsson A, Lalu MM, Li T, Loder EW, Mayo-Wilson E, McDonald S, McGuinness LA, Stewart LA, Thomas J, Tricco AC, Welch VA, Whiting P, Moher D. The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. BMJ. 2021: n71 [PMID: 33782057, https://doi.org/10.1136/bmj.n71] DOI: https://doi.org/10.1136/bmj.n71
8. Pollock M, Fernandes RM, Becker LA, Pieper D, Hartling L. Chapter V: overviews of reviews. Cochrane handbook for systematic reviews of interventions version. 2020:
9. OpenAi. GPT-4 Technical Report 2023. 2023.
10. McGowan J, Sampson M, Salzwedel DM, Cogo E, Foerster V, Lefebvre C. PRESS Peer Review of Electronic Search Strategies: 2015 Guideline Statement. J Clin Epidemiol. 2016: 40 [PMID: 27005575, https://doi.org/10.1016/j.jclinepi.2016.01.021] DOI: https://doi.org/10.1016/j.jclinepi.2016.01.021
11. J G, C L, P M, S R, I B, Woods L e. ISSG Search Filter Resource. 2006.
12. Toth B, Berek L, Gulacsi L, Pentek M, Zrubka Z. Automation of systematic reviews of biomedical literature: a scoping review of studies indexed in PubMed. Syst Rev. 2024: 174 [PMID: 38978132, https://doi.org/10.1186/s13643-024-02592-3] DOI: https://doi.org/10.1186/s13643-024-02592-3
13. de la Torre-López J, Ramírez A, Romero JR. Artificial intelligence to automate the systematic review of scientific literature. Computing. 2023: 2171 https://doi.org/10.1007/s00607-023-01181-x] DOI: https://doi.org/10.1007/s00607-023-01181-x
14. Schmidt L, Mohamed S, Meader N, Bacardit J, Craig D. Automated data analysis of unstructured grey literature in health research: A mapping review. Res Synth Methods. 2024: 178 [PMID: 38115736, https://doi.org/10.1002/jrsm.1692] DOI: https://doi.org/10.1101/2023.06.29.23291656
15. Schmidt L, Finnerty Mutlu AN, Elmore R, Olorisade BK, Thomas J, Higgins JPT. Data extraction methods for systematic review (semi)automation: Update of a living systematic review. F1000Res. 2021: 401 [PMID: 34408850, https://doi.org/10.12688/f1000research.51117.3] DOI: https://doi.org/10.12688/f1000research.51117.1
16. Yao X, Kumar MV, Su E, Flores Miranda A, Saha A, Sussman J. Evaluating the efficacy of artificial intelligence tools for the automation of systematic reviews in cancer research: A systematic review. Cancer Epidemiol. 2024: 102511 [PMID: 38071872, https://doi.org/10.1016/j.canep.2023.102511] DOI: https://doi.org/10.1016/j.canep.2023.102511
17. Doneva SE, Qin S, Sick B, Ellendorff T, Goldman J-P, Schneider G, Ineichen BV. Large language models to process, analyze, and synthesize biomedical texts: a scoping review. Discover Artificial Intelligence. 2024: https://doi.org/10.1007/s44163-024-00197-2] DOI: https://doi.org/10.1101/2024.04.19.588095
18. Santos AOD, da Silva ES, Couto LM, Reis GVL, Belo VS. The use of artificial intelligence for automating or semi-automating biomedical literature analyses: A scoping review. J Biomed Inform. 2023: 104389 [PMID: 37187321, https://doi.org/10.1016/j.jbi.2023.104389] DOI: https://doi.org/10.1016/j.jbi.2023.104389
Data Availability Statement
No new datasets were generated or analyzed in this umbrella review. All data supporting the findings are derived from previously published sources and are presented within the article and its supplementary materials.
License
Copyright (c) 2025 Maria Rizzo, Omar Irfan
This work is licensed under a Creative Commons Attribution 4.0 International License.
Authors retain copyright in their work and grant the publisher a non-exclusive right of first publication.
Downloads
Published
Issue
Section
How to Cite
Article history
- Received
- 2 Aug 2025
- Received in revised form
- 26 Nov 2025
- Accepted
- 7 Dec 2025
- Published
- 21 Dec 2025
