OPTIMIZING SCIENTIFIC VALUE: SMART AND SYSTEMATIC APPROACHES TO MEDICAL PUBLICATIONS

Transcription

1 11TH ANNUAL MEETING OF ISMPP OPTIMIZING SCIENTIFIC VALUE: SMART AND SYSTEMATIC APPROACHES TO MEDICAL PUBLICATIONS April 27 29, 2015 Hyatt Regency Crystal City Arlington, VA, USA 11TH ANNUAL MEETING OF ISMPP 1

2 11TH ANNUAL MEETING OF ISMPP Innovation in Metrics: Capturing the full impact of publications 11TH ANNUAL MEETING OF ISMPP 2

3 11TH ANNUAL MEETING OF ISMPP 3

4 Key questions in metrics What do you want to measure? The performance of: My agency (quality of service, cost/value) My team (quality, efficiency, productivity) Overall publication plan (volume, reach, understanding, impact) Individual publications (reach, understanding, impact) Why do you want to measure it? What is the best way to measure it? What are you going to do with that information? 11TH ANNUAL MEETING OF ISMPP 4

5 Key questions in metrics What do you want to measure? Why do you want to measure it? To show how I performing against comparators To understand if I should be doing things differently To defend spend/budget and demonstrate return on investment What is the best way to measure it? What are you going to do with that information? 11TH ANNUAL MEETING OF ISMPP 5

6 Key questions in metrics What do you want to measure? Why do you want to measure it? What is the best way to measure it? What are the best components of a publication analysis? The impact factor measures the success of the journal, not the impact of the article; what article level metrics are important? How do / can we measure the impact of congress activities as well journal publications? What are you going to do with that information? 11TH ANNUAL MEETING OF ISMPP 6

7 Key questions in metrics What do you want to measure? Why do you want to measure it? What is the best way to measure it? What are you going to do with that information? Defend (retrospective) Apply (prospective) 11TH ANNUAL MEETING OF ISMPP 7

8 NOT EVERYTHING THAT CAN BE COUNTED COUNTS, AND NOT EVERYTHING THAT COUNTS CAN BE COUNTED Jane Grimson. In Blockmans W et al, 2014 Bibliometrics: Use and Abuse in the Review of Research Performance 11TH ANNUAL MEETING OF ISMPP 8

9 11TH ANNUAL MEETING OF ISMPP Publication Metrics: Practical Application and Experience 11TH ANNUAL MEETING OF ISMPP 9

10 INTRODUCTIONS Moderator Sarah Feeney, BMedSc Head of Scientific Direction, Complete Medical Communications Panel Bhakti Kshatriya, PharmD Global Scientific Communications, Novartis Pharmaceuticals Corporation Tom Rees, MSc, PhD, CMPP TM Scientific Strategy Advisor, PAREXEL International Mike Buschman Co-Founder, Plum Analytics 11TH ANNUAL MEETING OF ISMPP 10

11 DISCLAIMER The opinions expressed in this presentation are those of the faculty and do not necessarily represent the opinions of their employers or ISMPP. Neither the faculty nor the facilitator have received honoraria for their participation in this workshop. 11TH ANNUAL MEETING OF ISMPP 11

12 OBJECTIVES Understand the process of developing publication plan metrics Be conversant in the various types of newer publication metrics Be educated on which metrics are of most importance to various stakeholders 11TH ANNUAL MEETING OF ISMPP 12

13 11TH ANNUAL MEETING OF ISMPP PUBLICATION METRICS PHARMA INDUSTRY PERSPECTIVE Bhakti Kshatriya, PharmD Global Scientific Communications Novartis Pharmaceuticals Corporation 11TH ANNUAL MEETING OF ISMPP 13

14 DISCLAIMER This presentation is made at the request of International Society for the Medical Publication Professionals (ISMPP). The presenter, although an employee of Novartis Pharmaceuticals Corporation, is presenting on her own behalf. The information provided and any views expressed during this presentation are those of the presenter and may not be attributed to Novartis Pharmaceuticals Corporation. 11TH ANNUAL MEETING OF ISMPP 14

15 WHY ESTABLISH METRICS? Establish common goals among team members and management Helps prioritize publication activities Facilitate discussion of new publication concepts Does concept align with goals? Help ensure appropriate resources for planned publication activities Support requests for new budget and/or head count Measure progress and identify areas for improvement 11TH ANNUAL MEETING OF ISMPP 15

16 CREATING METRICS 11TH ANNUAL MEETING OF ISMPP 16

17 FACTORS AFFECTING METRICS Internal Factors Product life cycle SOPs & policies Pub team effectiveness Pub team resources Data generation Data analyses Content generation (authors, writers, contributors, budget) External Factors Number of competitors Author perceptions Policies at author's institution New regulations Evolving publication guidelines and journal policies 11TH ANNUAL MEETING OF ISMPP 17

18 Publication Metrics Volume Timing Quality Topics ~ literature gap analysis 11TH ANNUAL MEETING OF ISMPP 18

19 PUBLICATION METRICS: VOLUME Number of manuscripts submitted & published Number of abstracts/posters/oral presentations at key scientific meetings Planned vs actual priority submissions/ acceptances Number of abstracts/manuscripts to regional meetings/journals 11TH ANNUAL MEETING OF ISMPP 19

20 PUBLICATION METRICS: VOLUME Number of Manuscripts Submitted 200 Primary/secondary manuscripts Review articles All Other (editorial, short reports, etc.) Q1 Q2 Q3 Q4 Metric can also be reported by drug/indication Similar metric can be analyzed for number of manuscripts published 11TH ANNUAL MEETING OF ISMPP 20

21 PUBLICATION METRICS: VOLUME Volume by Type of Publication (journal & congress) Manuscripts & Review Articles Abstracts Posters/Oral All Other YTD 11TH ANNUAL MEETING OF ISMPP 21

22 PUBLICATION METRICS: PLAN VS ACTUAL Number of Publications Planned Submissions Actual Submission per Plan Actual Submission not Planned for Quarter 0 Planned Actual Planned Actual Planned Actual Planned Actual Q1 Q2 Q3 Q4 11TH ANNUAL MEETING OF ISMPP 22

23 PUBLICATION METRICS: TIMING FDA and EudraCT guidelines Timing of publications for communication of clinical trial results Life cycle management May set very specific and/or accelerated timelines for key manuscripts (e.g., new indications, etc.) Timelines When measuring timelines, need to define start and end date of publication upfront (e.g., end date could be submit, accept, publish) 11TH ANNUAL MEETING OF ISMPP 23

24 MANUSCRIPT SUBMISSION TIMELINESS Drug A Drug B Drug C Weeks 11TH ANNUAL MEETING OF ISMPP 24

25 PUBLICATION METRICS: QUALITY Impact factor (for key publications) Visibility of journal or article Acceptance in first-choice journal Number of times the article or scientific statement is cited in other journal articles 11TH ANNUAL MEETING OF ISMPP 25

26 PUBLICATION METRICS: JOURNALS Journal values Lancet Am J Med Sci Diabet Med Diabetes Diabetes Care Diabetologia Am J Med Curr Med Res Opin Drug A Drug B Drug C 3 2 Diabetes, Obesity & Metabolism Diabetes Res Clin Pract Diabetic Medicine Endocrinology Endocr J Int J Clin Pract Postgraduate Medicine Ann Pharmacotherapy Best Practice Res Clin Endo Metab Clin Ther Diabetes Tech Ther Endocr Pract J Am Pharm Assoc J Clin Pharmacol J Manage Care Pharm Transplant International Transplantation Can J Clin Pharmacol Cardiovascular Diabetology Drugs Drugs Today Expert Opin Investig Drugs Expert Opin Pharmacother Horm Metab Res. Int J Clin Pharmacother Leuk Rec Mol Cell Endo Pharmacotherapy Vasc Health Risk Manag Prescrire Int 1 11TH ANNUAL MEETING OF ISMPP 26

27 ACCEPTANCE RATE COMPARED WITH JOURNAL AVERAGE Number of Articles Overall Journal Acceptance Rate Product-Specific Acceptance Rate ( ) 10 0 BJU International European Urology International Journal of Urology International Urogynecology J Journal of Sexual Medicine Neurourology & Urodynamics The Journal of Urology Urology 11TH ANNUAL MEETING OF ISMPP 27

28 EMERGING PUBLICATION METRICS Congress activities QR codes number of downloads Journal articles Online journals Number of downloads Number of clicks through to the article Number of page views Number of unique visitors Article appearing in the top 25 articles Specific to journal/publisher Number of times cited (Scopus) Novel and innovative ways of presenting publications 11TH ANNUAL MEETING OF ISMPP 28

29 SUMMARY Metrics vary and there are more than one that may apply to a given situation Metrics are dependent upon a product s life cycle Metrics should have purpose and measure success/effectiveness of the publication plan or team performance Emerging metrics may provide new opportunities for measure of publication reach & quality 11TH ANNUAL MEETING OF ISMPP 29

30 11TH ANNUAL MEETING OF ISMPP UNDERSTANDING MODERN IMPACT METRICS Mike Buschman Co-Founder, Plum Analytics April 29, TH ANNUAL MEETING OF ISMPP 30

31 Altmetrics? Article Level Metrics? 11TH ANNUAL MEETING OF ISMPP 31

32 Our Approach to Metrics Citations Usage Captures ALLmetrics Mentions Social Media 11TH ANNUAL MEETING OF ISMPP 32

33 Journal citations are the current state of 33 Scholarly Measure Journal Impact Factor TH ANNUAL MEETING OF ISMPP 33

34 Samples of PlumX Metrics Sources Amazon Bit.ly CrossRef Delicious Dryad dspace EBSCO eprints Facebook figshare Github Goodreads Google+ Medwave Mendeley PLOS PubMed Central Reddit Research Blogging Scopus SlideShare SourceForge Stack Overflow Twitter USPTO Vimeo Wikipedia Worldcat (OCLC) YouTube 11TH ANNUAL MEETING OF ISMPP 34

35 All Types of Research Artifacts Articles Blog posts Book chapters Books Cases Clinical Trials Conference Papers Data Sets Figures Grants Interviews Letters Media Patents Posters Presentations Reports Source Code Theses / Dissertations Videos Web Pages 11TH ANNUAL MEETING OF ISMPP 35

36 Metrics Categories USAGE (clicks, downloads, views, library holdings, video plays) CAPTURES (bookmarks, code forks, favorites, readers, watchers) MENTIONS (blog posts, comments, reviews, Wikipedia links) SOCIAL MEDIA (+1s, likes, shares, tweets) CITATIONS (PubMed Central, Scopus, patents) 11TH ANNUAL MEETING OF ISMPP 36

37 clicks, downloads, views, library holdings, USAGE video plays Is anyone reading our work? Did anyone watch our videos? Usage is the #1 stat researchers want to know after Citation counts PlumX is the only product that includes Usage 11TH ANNUAL MEETING OF ISMPP 37

38 bookmarks, favorites, readers, watchers CAPTURES Captures indicate that someone wants to come back to the work Early indicator of future citations 11TH ANNUAL MEETING OF ISMPP 38

39 blog posts, comments, reviews, MENTIONS Wikipedia links This category measures people truly engaging with your research Automatically uncover the conversations about your research Discover feedback, opinions, etc. 11TH ANNUAL MEETING OF ISMPP 39

40 +1s, likes, shares, tweets SOCIAL MEDIA Social media measures how well a researcher is promoting their work This is especially important for early career researchers to measure and understand Track the buzz and attention around your research output 11TH ANNUAL MEETING OF ISMPP 40

41 PubMed Central, Scopus, patents CITATIONS Citations still a standard of long-term impact Including citations enables side-by-side analysis with other metrics categories 11TH ANNUAL MEETING OF ISMPP 41

42 5 Categories in Action Example Article Article level metrics are the building blocks for PlumX 11TH ANNUAL MEETING OF ISMPP 42

43 2015 Article 11TH ANNUAL MEETING OF ISMPP 43

44 2014 Article 11TH ANNUAL MEETING OF ISMPP 44

45 2013 Article 11TH ANNUAL MEETING OF ISMPP 45

46 Researcher Profile 11TH ANNUAL MEETING OF ISMPP 46

47 Analytics document sets and researchers 11TH ANNUAL MEETING OF ISMPP 47

48 Journal Comparisons 11TH ANNUAL MEETING OF ISMPP 48

49 THANK YOU! Mike Buschman 11TH ANNUAL MEETING OF ISMPP 49

50 11TH ANNUAL MEETING OF ISMPP INTERPRETING ARTICLE METRICS Tom Rees, MSc, PhD, CMPP TM Scientific Strategy Advisor, PAREXEL International 11TH ANNUAL MEETING OF ISMPP 50

51 Why do we need metrics? What do we want to achieve Have we achieved it? ( and what are the subtle influences on metrics that could trip us up?) 11TH ANNUAL MEETING OF ISMPP 51

52 A framework for thinking about metrics Allen HG, et al. PloS one 8.7 (2013): e TH ANNUAL MEETING OF ISMPP 52

53 The best articles achieve higher metrics Bornmann, Lutz (2015): figshare. Retrieved 20:54, Apr 18, 2015 (GMT) 11TH ANNUAL MEETING OF ISMPP 53

54 But the type of article influences metrics differently Bornmann, Lutz (2015): figshare. Retrieved 20:54, Apr 18, 2015 (GMT) 11TH ANNUAL MEETING OF ISMPP 54

55 Different topics are picked up differently Mendeley Twitter Costas R, et al.(2015):. figshare. Retrieved 20:04, Apr 19, 2015 (GMT) 11TH ANNUAL MEETING OF ISMPP 55

56 Document characteristics affect metrics differently Correlation (r-squared) Citations Blog posts Tweets News Pages References Title length Countries Haustein S, et al. PLoS ONE 2015;0(3): e TH ANNUAL MEETING OF ISMPP 56

57 Open access increases scholarly activity more than social media Alhoori, Hamed, et al. "On the Relationship between Open Access and Altmetrics." iconference 2015 Proceedings (2015). 11TH ANNUAL MEETING OF ISMPP 57

58 Even within open access, choice of journal influences metrics differently Rees TJ, et al. Poster presentation ISMPP 8AM TH ANNUAL MEETING OF ISMPP 58

59 Video abstracts don t affect views but does that matter? Rees TJ, et al. Poster presentation ISMPP 11AM TH ANNUAL MEETING OF ISMPP 59

60 Few researchers currently use social media to discover recommended research papers % of service users % of full sample* Twitter Facebook Frontiers Academia.edu ResearchGate LinkedIn BioMedExperts Google+ Mendeley 0% 20% 40% 60% 0% 20% 40% 60% Rees TJ, et al. Oral presentation ISMPP European Meeting 2014 * Extrapolation 11TH ANNUAL MEETING OF ISMPP 60

61 And social media is not just a youngperson thing % Age (years) (NPG), Nature Publishing Group (2014): NPG 2014 Social Networks survey. figshare. 11:36, Apr 24, 2015 (GMT) 11TH ANNUAL MEETING OF ISMPP 61

62 All that tweets is not gold 11TH ANNUAL MEETING OF ISMPP 62

63 Dam that Social Networking 11TH ANNUAL MEETING OF ISMPP 63

64 People are consuming this information! Feeny S, et al. ISMPP 10AM TH ANNUAL MEETING OF ISMPP 64

65 Conclusion Evaluating publication impact is complex, and so requires multiple measures Understanding what different metrics indicate is critical to their interpretation We are still limited to measuring what we can, rather than what we want to Remember, Activity Impact! 11TH ANNUAL MEETING OF ISMPP 65

66 11TH ANNUAL MEETING OF ISMPP Innovation in Metrics: Capturing the full impact of publications 11TH ANNUAL MEETING OF ISMPP 66

67 11TH ANNUAL MEETING OF ISMPP OPTIMIZING SCIENTIFIC VALUE: SMART AND SYSTEMATIC APPROACHES TO MEDICAL PUBLICATIONS April 27 29, 2015 Hyatt Regency Crystal City Arlington, VA, USA 11TH ANNUAL MEETING OF ISMPP 67