Measuring research in the Big Data Era: the evolution of Performance Measurement Systems in the Italian Teaching Hospitals

Abstract Background In the healthcare system, Teaching Hospitals(THs) not only provide care, but also train healthcare professionals and carry out research activities. Research is a fundamental pillar of THs’ mission and relevant for the healthcare system monitored by Performance Evaluation Systems. Research activities can be measured using citation index services and this paper highlights differences between two services based on bibliometrics, describes opportunities and risks when performance indicators rely on data collected, controlled and validated by external services and discusses the possible impact on health policy at a system and provider level. Methods A bibliometric analysis was done on data between 2014-2016 from ISI Web of Science and Scopus of 18.255 physicians working in 26 Italian THs. Quantity was defined as the number of publications and quality as Impact Factor or Field-Weighted Citation Impact. Results Overall, 41.233 and 66.409 documents were extracted from respectively ISI Web of Science and Scopus. While benchmarking results, significant differences in ranked position both in metrics emerged. Discussion Utilizing secondary data sources to measure research activities of THs allows benchmarking at an (inter)national level and overcoming self-referment. To utilize indicators for multiple governance purposes at the system and provider level, indicators need to be profoundly understood, require formalizations in data validation, internal analysis and a sharing process among health professionals, management and policymakers. Abbreviations IRPESInter-Regional-Performance Evolution systemLHALocal Health AuthorityPESPerformance evaluation systemTHTeaching hospitalWoSISI Web of Science Keywords Teaching hospitalsResearch productivityPerformance evaluationImpact factorField-Weighted Citation Impact