SpringerLink
Log in

Policy citations of scientometric articles: an altmetric study

  • Published:
Scientometrics Aims and scope Submit manuscript

Abstract

Policy citations are considered as one of the important indicators of the societal impact of research. Scientometrics is a field that, among other goals, focus on contributing to science policy, so the presence of scientometric researches in policy documents become important. Accordingly, this study aims to measure the impact of scientometric researches on policy by examining the mentions of scientometric articles in policy documents. The dataset used in this study includes 5525 scientometric articles indexed in Web of Science between 2013 and 2022. The Overton database were used to collect policy citations. The results showed that out of 5525 scientometric articles, 921 articles (16.67%) were cited at least once in policy documents. Additionally, older articles were cited more frequently than recent ones in policy documents. Scientometrics Journal ranked first in terms of the number of articles being cited in policy documents, while Research Policy and Research Evaluation Journals ranked first and second, respectively, in terms of coverage, density, and intensity. Subject analysis of the cited articles in policy documents showed that articles on national/international scholar collaborations, scholar productivity/scholar performance, and funding and sponsorship were cited more frequently in policy documents. Finally, Open Access articles were cited more frequently than non-open access articles in policy documents. However, there was not significant difference between policy citations of multi-authored and sing-authored articles. Overall, policy citations of scientometric articles were fair compared to other fields, and for greater impact of this field on policy, publishing open access, and greater attention to the topics identified in this study can be helpful.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
EUR 32.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or Ebook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1

Similar content being viewed by others

Notes

  1. The previous name of the journal was Journal of the American Society for Information Science and Technology which was considered in the search strategies.

References

  • Adie, E. (2020. Do open journals get cited more in government policy? In Septentrio Conference Series (No. 4).

  • Aguillo, I. F. (2020). Altmetrics of the open access institutional repositories: A webometrics approach. Scientometrics, 123(3), 1181–1192. https://doi.org/10.1007/s11192-020-03424-6

    Article  Google Scholar 

  • Bornmann, L. (2013). What is societal impact of research and how can it be assessed? A literature survey. Journal of the American Society for Information Science and Technology, 64(2), 217–233.

    Article  Google Scholar 

  • Bornmann, L. (2014). Validity of altmetrics data for measuring societal impact: A study using data from Altmetric and F1000Prime. Journal of Informetrics, 8, 935–950. https://doi.org/10.1016/j.joi.2014.09.007

    Article  Google Scholar 

  • Bornmann, L., Haunschild, R., Boyack, K., Marx, W., & Minx, J. C. (2022). How relevant is climate change research for climate change policy? An empirical analysis based on Overton data. PLoS ONE, 17(9), e0274693. https://doi.org/10.1371/journal.pone.0274693

    Article  Google Scholar 

  • Bornmann, L., Haunschild, R., & Marx, W. (2016). Policy documents as sources for measuring societal impact: How often is climate change research mentioned in policy-related documents? Scientometrics, 109(3), 1477–1495.

    Article  Google Scholar 

  • Campbell, S. E., & Simberloff, D. (2022). Forty years of invasion research: More papers, more collaboration... bigger impact? NeoBiota, 75, 57–77.

    Article  Google Scholar 

  • Drongstrup, D., Malik, S., Aljohani, N. R., Alelyani, S., Safder, I., & Hassan, S.-U. (2020). Can social media usage of scientific literature predict journal indices of AJG, SNIP and JCR? An Altmetric Study of Economics. Scientometrics, 125(2), 1541–1558. https://doi.org/10.1007/s11192-020-03613-3

    Article  Google Scholar 

  • Fang, Z., Dudek, J., Noyons, E., & Costas, R. (2020). Science cited in policy documents: Evidence from the Overton database. In Altmetrics conference.

  • Ferreras-Fernández, T., García-Penalvo, F. J., & Merlo-Vega, J. A. (2015). Open access repositories as channel of publication scientific grey literature. Paper presented at the Proceedings of the 3rd International Conference on Technological Ecosystems for Enhancing Multiculturality, Porto, Portugal.

  • Haunschild, R., & Bornmann, L. (2017). How many scientific papers are mentioned in policy-related documents? An empirical investigation using web of science and altmetric data. Scientometrics, 110, 1209–1216. https://doi.org/10.1007/s11192-016-2237-2

    Article  Google Scholar 

  • Haustein, S., Costas, R., & Larivière, V. (2015). Characterizing social media metrics of scholarly papers: The effect of document properties and collaboration patterns. PLoS ONE, 10(3), e0120495. https://doi.org/10.1371/journal.pone.0120495

    Article  Google Scholar 

  • Hicks, D., & Isett, K. R. (2020). Powerful numbers: Exemplary quantitative studies of science that had policy impact. Quantitative Science Studies, 1(3), 969–982. https://doi.org/10.1162/qss_a_00060

    Article  Google Scholar 

  • Hood, W., & Wilson, C. (2001). The literature of bibliometrics, scientometrics, and informetrics. Scientometrics, 52(2), 291–314.

    Article  Google Scholar 

  • Huang, Z., Zong, Q., & Ji, X. (2022). The associations between scientific collaborations of LIS research and its policy impact. Scientometrics, 127(11), 6453–6470.

    Article  Google Scholar 

  • Joly, P. B., Gaunand, A., Colinet, L., Larédo, P., Lemarié, S., & Matt, M. (2015). ASIRPA: A comprehensive theory-based approach to assessing the societal impacts of a research organization. Research Evaluation, 24(4), 440–453.

    Article  Google Scholar 

  • Kay, A. (2011). Evidence-based policy-making: The elusive search for rational public administration. Australian Journal of Public Administration, 70(3), 236–245. https://doi.org/10.1111/j.1467-8500.2011.00728.x

    Article  Google Scholar 

  • Khasseh, A. A., Soheili, F., Moghaddam, H. S., & Chelak, A. M. (2017). Intellectual structure of knowledge in iMetrics: A co-word analysis. Information Processing & Management, 53(3), 705–720.

    Article  Google Scholar 

  • Lemke, S., Witthake, A., & Peters, I. (2022). Altmetrics for German medical research: What leads to research articles achieving policy impact? 26th International Conference on Science, Technology and Innovation Indicators (STI 2022), Granada, Spain. https://doi.org/10.5281/ZENODO.6645109

  • Lui, J. (2014). New Source Alert: Policy Documents. Altmetric Blog. Retrieved https://www.altmetric.com/blog/new-source-alert-policy-documents/

  • Maleki, A., & Holmberg, K. (2022). Comparing coverage of policy citations to scientific publications in Overton and Altmetric. com: Case study of Finnish research organizations in Social Science. Informaatiotutkimus, 41(2–3), 92–96.

    Article  Google Scholar 

  • Mansourian, Y. (2010). Fifty research axes in scientometric studies. Kitab Mah Kliat, 54, 64–71. [in Persian].

    Google Scholar 

  • Mingers, J., & Leydesdorff, L. (2015). A review of theory and practice in scientometrics. European Journal of Operational Research, 246(1), 1–19.

    Article  Google Scholar 

  • Mryglod, O., Nazarovets, S., & Kozmenko, S. (2021). Universal and specific features of Ukrainian economic research: Publication analysis based on Crossref data. Scientometrics, 126(9), 8187–8203. https://doi.org/10.1007/s11192-021-04079-7

    Article  Google Scholar 

  • Musa, T. H., Li, W., Kawuki, J., & Wei, P. (2021). The 100 top-cited articles on scrub typhus: A bibliometric analysis. Osong Public Health and Research Perspectives, 12(2), 126–135.

    Article  Google Scholar 

  • Nightingale, P., & Scott, A. (2007). Peer review and the relevance gap: Ten suggestions for policy-makers. Science and Public Policy, 34(8), 543–553.

    Article  Google Scholar 

  • Pinheiro, H., Vignola-Gagné, E., & Campbell, D. (2021). A large-scale validation of the relationship between cross-disciplinary research and its uptake in policy-related documents, using the novel Overton altmetrics database. Quantitative Science Studies, 2(2), 616–642. https://doi.org/10.1162/qss_a_00137

    Article  Google Scholar 

  • Pohl, H. (2021). Internationalisation, innovation, and academic–corporate co-publications. Scientometrics, 126(2), 1329–1358. https://doi.org/10.1007/s11192-020-03799-6

    Article  Google Scholar 

  • Regan, Á., & Henchion, M. (2019). Making sense of altmetrics: The perceived threats and opportunities for academic identity. Science and Public Policy, 46(4), 479–489. https://doi.org/10.1093/scipol/scz001

    Article  Google Scholar 

  • Shrivastava, R., & Mahajan, P. (2023). Altmetrics and their relationship with citation counts: A case of journal articles in physics. Global Knowledge, Memory and Communication, 72(4/5), 391–407.

    Article  Google Scholar 

  • Szomszor, M., & Adie, E. (2022). Overton: A bibliometric database of policy document citations. Quantitative Science Studies, 3(3), 624–650. https://doi.org/10.1162/qss_a_00204

    Article  Google Scholar 

  • Tague-Sutcliffe, J. M. (1992). An introduction to informetrics. Information Processing & Management, 28, 1–3.

    Article  Google Scholar 

  • Tattersall, A., & Carroll, C. (2018). What can Altmetric. com tell us about policy citations of research? An analysis of Altmetric. com data for research articles from the University of Sheffield. Frontiers in Research Metrics and Analytics, 2, 9.

    Article  Google Scholar 

  • Taylor, M. (2020). An altmetric attention advantage for open access books in the humanities and social sciences. Scientometrics, 125(3), 2523–2543.

    Article  Google Scholar 

  • Thelwall, M., Kousha, K., Dinsmore, A., & Dolby, K. (2016). Alternative metric indicators for funding scheme evaluations. Aslib Journal of Information Management, 68, 2–18. https://doi.org/10.1108/AJIM-09-2015-0146

    Article  Google Scholar 

  • Van Raan, A. (1997). Scientometrics: State-of-the-art. Scientometrics, 38(1), 205–218.

    Article  Google Scholar 

  • Xu, C., & Zong, Q. (2023). The effects of international research collaboration on the policy impact of research: A causal inference drawing on the journal Lancet. Journal of Information Science. https://doi.org/10.1177/01655515231174381

    Article  Google Scholar 

  • Yin, Y., Gao, J., Jones, B. F., & Wang, D. (2021). Coevolution of policy and science during the pandemic. Science, 371(6525), 128–130.

    Article  Google Scholar 

  • Yu, H., Murat, B., Li, J., & Li, L. (2023). How can policy document mentions to scholarly papers be interpreted? An analysis of the underlying mentioning process. Scientometrics. https://doi.org/10.1007/s11192-023-04826-y

    Article  Google Scholar 

  • Zong, Q., Huang, Z., & Huang, J. (2023). Can open access increase LIS research’s policy impact? Using regression analysis and causal inference. Scientometrics. https://doi.org/10.1007/s11192-023-04750-1

    Article  Google Scholar 

Download references

Acknowledgements

We would like to thank the Overton for granting us access to the Overton database.

Funding

This research did not receive any specific grant from funding agencies in the public, commercial, or not-for-profit sectors.

Author information

Authors and Affiliations

  1. University of Tabriz, Tabriz, East Azerbaijan, Iran

    Hashem Atapour, Robabeh Maddahi & Rasoul Zavaraqi

Authors
  1. Hashem Atapour
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Robabeh Maddahi
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Rasoul Zavaraqi
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Hashem Atapour.

Ethics declarations

Conflict of interest

The authors have no relevant financial or non-financial interests to disclose.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Atapour, H., Maddahi, R. & Zavaraqi, R. Policy citations of scientometric articles: an altmetric study. Scientometrics (2024). https://doi.org/10.1007/s11192-024-05091-3

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s11192-024-05091-3

Keywords

Search

Navigation