More than just numbers: Successes and challenges in meeting the demand for quantitative measures of research output

Transcription

1 More than just numbers: Successes and challenges in meeting the demand for quantitative measures of research output Thomson Reuters Understanding Federal R&D Impact through Research Assessment and Program Evaluation Al Berger March 19, 2013 NREL is a national laboratory of the U.S. Department of Energy, Office of Energy Efficiency and Renewable Energy, operated by the Alliance for Sustainable Energy, LLC.

2 2 National Renewable Energy Laboratory, Golden, CO Quick Facts National Centers: o National Bioenergy Center o National Center for Photovoltaics o National Wind Technology Center Research at NREL: o 13 main programmatic areas Employment: o 1,700 fulltime employees o 800 visiting researchers, interns, and contractors

3 3

4 Overview of Evaluation Methods for R&D Programs 14 Methods Peer Review/Expert Judgment Monitoring, Data Compilation, and Use of Indicators Bibliometric Methods Counts and Citation Analysis Bibliometric Methods Data Mining Bibliometrics Hotspot Patent Analysis Network Analysis Case Study Method Survey Method Benchmarking Method Technology Commercialization Tracking Method Benefit-Cost Case Study Econometric Methods Historical Tracing Method Spillover Analysis Using a Combination of Methods Source: Ruegg, R.; Jordan, G. (2007) Overview of Evaluation Methods for R&D Programs: A Directory of Evaluation Methods Relevant to Technology Development Programs (Draft). 4

5 Publication Metrics SharePoint Site 5

6 Are we publishing in high impact journals? What do we know about our impact? Do we offer institutional comparison? 6

7 Journal Impact Factors 7

8 Institutional Comparisons 8

9 Executive Dashboard 9

10 Executive Dashboard 10

11 Special Projects CdTe - Refined by Web of Science Categories: Total Papers Polish Acad Sci CNRS Russian Acad Sci Chinese Acad Sci Univ Wurzburg Univ Illinois Univ Grenoble Jilin Univ CNR CEA Natl Renewable Energy Lab Univ Durham Univ Hull USA (Army) Univ Warsaw CEN (France) Stanford Univ Tohoku Univ IPN Inst Politech Nacl Univ Hamburg Natl Acad Sci Ukraine Univ Freiburg Univ Tokyo Univ Autonoma Madrid CdTe - Refined by Web of Science Categories Total Papers 11

12 12 Special Projects CdTe - Refined by Web of Science Categories: Avg. Cites per Item Univ Hamburg CEN (France) Natl Renewable Energy Lab Stanford Univ Univ Freiburg Univ Grenoble Jilin Univ Univ Illinois Chinese Acad Sci CEA Univ Tokyo CNRS Univ Wurzburg Univ Hull Univ Durham Tohoku Univ USA (Army) Univ Autonoma Madrid CNR IPN Univ Warsaw Inst Politech Nacl Polish Acad Sci Russian Acad Sci Natl Acad Sci Ukraine CdTe - Refined by Web of Science Categories Avg. Cites per Item

13 13 Special Projects CdTe - Refined by Web of Science Categories: H Index Univ Hamburg Chinese Acad Sci Univ Illinois Univ Grenoble CEN (France) CNRS Natl Renewable Energy Lab Stanford Univ Univ Wurzburg Jilin Univ CEA Univ Freiburg Polish Acad Sci Univ Durham Univ Hull USA (Army) Univ Tokyo Russian Acad Sci CNR Tohoku Univ IPN Univ Warsaw Inst Politech Nacl Univ Autonoma Madrid Natl Acad Sci Ukraine CdTe - Refined by Web of Science Categories H Index

14 14 Challenges & Opportunities Customer Relations: Understanding what metrics are needed and how they will be used. The amount and diversity of data: Having the right tools at the right time. Valuation of non-traditional research products such as data sets and software.

15 Challenges & Opportunities New ways to think about how research output is measured. Looking beyond citation metrics, to: o Usage downloads, views o Peer review o Alternative Metrics (Altmetrics) links, bookmarks, conversations 15

16 16 Challenges & Opportunities Rapidly changing fields of research evaluation and scholarly communication. o Piwowar, Heather. Value all Research Products (Comment). Nature, Jan. 10, 2013, v. 493, p doi: /493159a o Clobridge, Abby. The Future of Open Access. Information Today, Jan. 2013, v. 30, #1, p b=bth&an= &site=ehost-live

17 NREL Library Informing Research, Advancing Discovery