Mapping the Evolution of Human-in-the-Loop Artificial Intelligence: A Bibliometric Analysis
Pages:
27-44Keywords:
human-in-the-loop, artificial intelligence, human–AI collaboration, bibliometric analysis, human oversight, responsible AIAbstract
Context: Human-in-the-loop (HITL) artificial intelligence has evolved from human-assisted model training toward broader human–AI collaboration, oversight, and decision-making. Its rapidly expanding and multidisciplinary literature, however, remains fragmented across technological and application domains, creating a need for systematic mapping of the field.
Aim: This study maps the evolution, scholarly impact, geographical and institutional structure, collaboration patterns, and conceptual development of HITL AI research from 2016 to 2026.
Approach: A bibliometric analysis was conducted on 2,071 Scopus-indexed publications. Bibliometrix and its Biblioshiny interface were used for bibliometric performance analysis, and VOSviewer was used for keyword co-occurrence mapping and visualization.
Key Findings: HITL AI research recorded a 55.9% annual growth rate and involved 8,319 authors publishing across 1,029 sources. The United States and China led publication output. Keyword analysis identified 462 keywords organized into five clusters, with prominent themes including deep learning, reinforcement learning, generative AI, decision support, responsible AI, ethics, and human oversight.
Interpretation: The findings indicate that HITL AI is developing into an increasingly multidisciplinary field in which human participation extends beyond model training toward collaboration, oversight, and accountable AI deployment.
Contribution: This study provides an integrated bibliometric mapping of HITL AI that consolidates its growth, scholarly impact, leading contributors, collaboration patterns, and conceptual structure while identifying emerging directions for future research.
References
Amaliah, N. R., Tjahjono, B., & Palade, V. (2025). Human-in-the-loop XAI for predictive maintenance: A systematic review of interactive systems and their effectiveness in maintenance decision-making. Electronics, 14(17), 3384. https://doi.org/10.3390/electronics14173384
Aria, M., & Cuccurullo, C. (2017). bibliometrix: An R-tool for comprehensive science mapping analysis. Journal of Informetrics, 11(4), 959–975. https://doi.org/10.1016/j.joi.2017.08.007
Chen, H., Li, S., Fan, J., Duan, A., Yang, C., Navarro-Alarcon, D., & Zheng, P. (2025). Human-in-the-loop robot learning for smart manufacturing: A human-centric perspective. IEEE Transactions on Automation Science and Engineering, 22, 11062–11086. https://doi.org/10.1109/TASE.2025.3528051
Enarsson, T., Enqvist, L., & Naarttijärvi, M. (2022). Approaching the human in the loop – legal perspectives on hybrid human/algorithmic decision-making in three contexts. Information & Communications Technology Law, 31(1), 123–153. https://doi.org/10.1080/13600834.2021.1958860
Gómez-Carmona, O., Casado-Mansilla, D., López-de-Ipiña, D., & García-Zubia, J. (2024). Human-in-the-loop machine learning: Reconceptualizing the role of the user in interactive approaches. Internet of Things, 25, 101048. https://doi.org/10.1016/j.iot.2023.101048
Hanafi, M., Katsis, Y., Jindal, I., & Popa, L. (2022). A comparative analysis between human-in-the-loop systems and large language models for pattern extraction tasks. In Proceedings of the Fourth Workshop on Data Science with Human-in-the-Loop (Language Advances) (pp. 43–50). Association for Computational Linguistics. https://doi.org/10.18653/v1/2022.dash-1.7
Holzinger, A. (2016). Interactive machine learning for health informatics: When do we need the human-in-the-loop? Brain Informatics, 3(2), 119–131. https://doi.org/10.1007/s40708-016-0042-6
Lazaros, K., Vrahatis, A. G., & Kotsiantis, S. (2026). Human-in-the-loop artificial intelligence: A systematic review of concepts, methods, and applications. Entropy, 28(4), 377. https://doi.org/10.3390/e28040377
Liu, H., Nasiriany, S., Zhang, L., Bao, Z., & Zhu, Y. (2025). Robot learning on the job: Human-in-the-loop autonomy and learning during deployment. The International Journal of Robotics Research, 44(10–11), 1727–1742. https://doi.org/10.1177/02783649241273901
Liu, H. K., Tang, M., & Collard, A. S. J. (2025). Hybrid intelligence for the public sector: A bibliometric analysis of artificial intelligence and crowd intelligence. Government Information Quarterly, 42(1), 102006. https://doi.org/10.1016/j.giq.2024.102006
Memarian, B., & Doleck, T. (2024). Human-in-the-loop in artificial intelligence in education: A review and entity-relationship (ER) analysis. Computers in Human Behavior: Artificial Humans, 2(1), 100053. https://doi.org/10.1016/j.chbah.2024.100053
Mosqueira-Rey, E., Hernández-Pereira, E., Alonso-Ríos, D., Bobes-Bascarán, J., & Fernández-Leal, Á. (2023). Human-in-the-loop machine learning: A state of the art. Artificial Intelligence Review, 56(4), 3005–3054. https://doi.org/10.1007/s10462-022-10246-w
Olawade, D. B., Plabon, S. B., Ojo, A., Ogunbona, M. A., Makanjuola, B. D., & Olasilola, O. R. (2026). Human in the loop artificial intelligence in healthcare: Applications, outcomes, and implementation challenges. International Journal of Medical Informatics, 213, 106362. https://doi.org/10.1016/j.ijmedinf.2026.106362
Retzlaff, C. O., Das, S., Wayllace, C., Mousavi, P., Afshari, M., Yang, T., Saranti, A., Angerschmid, A., Taylor, M. E., & Holzinger, A. (2024). Human-in-the-loop reinforcement learning: A survey and position on requirements, challenges, and opportunities. Journal of Artificial Intelligence Research, 79, 359–415. https://doi.org/10.1613/jair.1.15348
Salloch, S., & Eriksen, A. (2024). What are humans doing in the loop? Co-reasoning and practical judgment when using machine learning-driven decision aids. The American Journal of Bioethics, 24(9), 67–78. https://doi.org/10.1080/15265161.2024.2353800
Sele, D., & Chugunova, M. (2024). Putting a human in the loop: Increasing uptake, but decreasing accuracy of automated decision-making. PLOS ONE, 19(2), e0298037. https://doi.org/10.1371/journal.pone.0298037
Toumanidis, L., Kasnesis, P., Chatzigeorgiou, C., Feidakis, M., & Patrikakis, C. (2021). ActiveCrowds: A human-in-the-loop machine learning framework. In C. Frasson, K. Kabassi, & A. Voulodimos (Eds.), Novelties in intelligent digital systems: Proceedings of the 1st International Conference (NIDS 2021), Athens, Greece, September 30 – October 1, 2021 (pp. 176–184). IOS Press. https://doi.org/10.3233/FAIA210090
van Eck, N. J., & Waltman, L. (2010). Software survey: VOSviewer, a computer program for bibliometric mapping. Scientometrics, 84(2), 523–538. https://doi.org/10.1007/s11192-009-0146-3
Wu, X., Xiao, L., Sun, Y., Zhang, J., Ma, T., & He, L. (2022). A survey of human-in-the-loop for machine learning. Future Generation Computer Systems, 135, 364–381. https://doi.org/10.1016/j.future.2022.05.014
Downloads
Published
Data Availability Statement
The bibliographic data analyzed in this study were retrieved from the Scopus database on 26 August 2026. The analyzed data may be made available by the author upon reasonable request, subject to Scopus licensing and database-access conditions.
Issue
Section
License
Copyright (c) 2026 Asif Altaf (Author)
This work is licensed under a Creative Commons Attribution 4.0 International License.
Authors retain copyright in their work and grant the Journal of Scholarly Research and Metrics (JSRM) the right of first publication.
Articles published in JSRM are licensed under the Creative Commons Attribution 4.0 International License (CC BY 4.0). This license permits use, sharing, distribution, reproduction, and adaptation of the work in any medium or format, provided appropriate credit is given to the original author(s) and source, a link to the license is provided, and any changes made are indicated.