Journal Impact: Can we do better? Nicholas J. Talley, MD, PhD, FRACP Editor-in-Chief MJA

Transcription

1 Journal Impact: Can we do better? Nicholas J. Talley, MD, PhD, FRACP Editor-in-Chief MJA 1

2

3 Impact factor the flawed metric editors live and die by Flaws in the IF: Driven by a minority of the papers (exaggerates the underperformers!) Papers published in the first half of the year more influential on number Calculation by Thompson Scientific not highly reproducible Subject to inflation Can potentially be manipulated by editors e.g. self citations and WJG publish reviews in preference to original articles H index (reflecting number of publications and number of citations) applies to individual researchers not journals

4 Nature

5 What is a good impact factor? A debate In 2012, the Journal Citation Reports (JCR) database assigned impact factors to 8,411 journals Only 21 journal titles, or 0.2% of the journals tracked by JCR, had a 2012 one-year impact factor of 30. Only 158 journals had a 2012 one-year impact factor equal to or greater than 10 (top 1.9%); above 3 top 10%

6 Scholarly Impact and rigour is a springboard to wider impact Impact Factor is unlikely to be superseded as a primary measure of reputation, rigour or value in the immediate future, increasingly it is not a sufficient measure E.g. in 2008 the Australian NHMRC stated that it was not looking at Impact Factor alone as a measure of quality research output

7 Some other current impact measures

8 Scholarly impact and rigour is a springboard to wider impact What are some of the world s most successful medical journals doing to build on their traditional scholarly prestige and profile to extend their influence and success to scholars, clinicians, politicians and the lay reader? These Journals are actively committed to broadening their clinical and societal impact through a range of initiatives Can we learn from their example and methods?

9 What do the best Journals do? Ways of Leveraging Impact With a 2016 Impact Factor of NEJM is by far the most successful and highly cited medical journals globally The NEJM brand is a gold standard of legitimacy and it s societal impact is derived from this It generates enormous impact through the sheer scale and diversity of its readership It publishes world class research, attracts expert authors and peer reviewers: would you decline an invitation to peer review an article that may be published in the NEJM?

10 Ways of achieving Impact Scholarly Impact

11 Ways of achieving Scholarly Impact

12 Ways of leveraging Scholarly Impact The global power brand of scholarly medical publishing within and beyond the medical research and clinical community Beyond Impact Factor it measures and publicises a range of metrics for every article published, this is a common practice amongst medical journals Numerous articles attract around 20,000 unique page views online in it s first 7-15 days of publication!

13 Health System Change Impact- Influencing health system and health policy? The Lancet s 2016 Impact Factor is The Lancet has an excellent reputation for and capacity to influence debate about changes in health systems and in wider global and public health issues. It is explicitly committed to shaping and participating in these policy debates Aside from it s core weekly Journal and array of high quality specialty and open access Journals, The Lancet has set itself an ambitious target of publishing 15 new Journals in 5 years, on current indicators it is on track to achieve this A recent series has been launched that focuses on the health and health systems of specific nations. To date journals have been published on France and Israel with more to come

14 Health System Change Impact - Influencing health system and health policy? It is at times controversial, intentionally and unintentionally and this is a strategy to drive awareness and influence debate.

15 Form partnerships to publish important and emerging areas of medical science and public health Rockefeller Foundation-Lancet collaborations have given rise to 3 new open access new Journals, covering the interplay between health and the determinant s of health in our living and physical world (Inaugural editorial Lancet Planetary Health). The Lancet Public Health, The Lancet Global Health and in April 2017 The Lancet Planetary Health These are creating impact by informing policy and debate globally and in the case of Planetary Health it is playing a role in shaping the discipline itself

16 Expanding your impact by being accessible The Lancet is adept at simplifying complex public health issues for lay readers and the wider media. They excel at infographics This positions them very well for social media engagement, traditional media coverage and talking to politicians and political advisors

17

18 Translational Impact- improving and informing clinical practice Aside from being a globally renowned general medical scholarly journal it places a strong and quite practical emphasis on translating quality, cutting edge research into quality practice at the point of care. BMJ s stated purpose in it s Living our Values Report is: Patients come first Our overriding commitment is to help healthcare professionals do the best for patients and the public. A respect for patient safety, confidentiality and patient choice, underpins our culture, brand and behaviours How is the BMJ living this value and creating impact that reflects it?

19 Translational Impact- improving and informing clinical practice The BMJ launched it s patient partnership strategy and is the first medical journal to receive the Patients included Certificate. In the 1 st year the BMJ has over 300 patient reviewers and co-creators Authors or medical education articles are being asked to involve or co-produce certain articles with patients, and declare the nature of their involvement Requesting authors of research papers to disclose if or how they involved patients in setting the research question, the outcome measures, the design and implementation of the study, and the dissemination of results. Embedding patient review of papers in standard peer review processes Crucially appointing patients and patient advocates to their editorial board Appointing a patient editor who launched a new article series, What your Patient is Thinking in 2015 and issued The BMJ Patient Spotlight edition featuring patient co-created articles.

20 Translational Impact- improving and informing clinical practice Leveraging the BMJ brand and producing world class, globally applicable point of care tools and information- BMJ Best Practice

21 The MJA

22 How does the MJA seek to achieve Impact? As a general medical journal the MJA is top tier Journal (revised year Impact Factor of 4.0). We have a clear purpose and goals: 1. Maintain our editorial independence 2. Publish world class, evidence based research but also reviews and guidelines 3. Help translate evidence based research into improve clinical practice for all doctors (a large number of our regular readership are AMA members and practicing clinicians) and patients lives 4. Positively influence health system change and health policy 5. Improve public health by generating awareness, through our own medical Journalism, MJA InSight and by being part of the public debate through traditional and social media channels

23 Longstanding and highly regarded general medical journal The MJA celebrated its centenary in 2014 Distinguished history, e.g. published Barry Marshall s famous self-ingestion H. pylori experiment that paved the way to the Nobel prize antibiotics for peptic ulcer discovery of Q fever folate deficiency and neural tube defects rubella and foetal damage health care can harm you

24 All original research is fee-free and open access A hybrid journal Print (to 33,000 AMA members and subscribers) and online Authors do NOT pay to publish in the MJA All original research articles are available without charge on the MJA website (readers must register for access, except doctors and medical students; not limited to AMA members) Selected other articles are also open access; e.g., all articles in the Indigenous issue (published annually in July)

25 Rigorous editorial processes: Double-blind peer review Peer review is critical to value add of a medical journal (although under scrutiny and more research a must will need partnerships to succeed) research being planned Rigorous peer review at the MJA Peer review is double blinded reviewers do not know who the authors are until the paper is published All original research, reviews and perspectives peer-reviewed All original research undergoes formal statistical review All articles after peer review reviewed by the Journal editors, including a statistical editor, in a conference meeting format, twice/week ( hanging committee ) >85% rejection rate for original research

26 High impact on multiple metrics Impact is more than the impact factor and Altmetrics Need new ways to measure Journal impact High readership MJA and MJA InSight: 1.7m page views YTD! High press interest every issue The MJA influences government policy health departments (Federal and States) subscribe and read the Journal, Ministers (or at least their minders) pay attention Leader in indigenous health research and policy

27 2017 Google Analytics for MJA.com.au

28 Full editorial control over and the publishing (and my first edition ) The MJA has guaranteed editorial independence from the Australian Medical Association (AMA) There is not once instance in my tenure when anyone has even tried to influence editorial decisions (and I would not stand for it!) The editorial and structural editing process is all in house, including MJA staff who are expert structural editors with a strong research backgrounds Elsevier prints the Journal but has no other role in the process a completely hands off relationship

29 Alternatives to IF ranking of journals (source: University of Michigan Library) Eigenfactor.org A free and searchable database, journals listed in the Eigenfactor database may be searched by ISSN, publisher, year published, exact journal name or by using the Journal Citation Report ISI categories. The Eigenfactor site includes the Eigenfactor Motion Graph which is a way to view changes within scholarly publishing from 1997 based on journal Eigenfactor scores, article influence, journal price, self citation rate and other metrics Centre for Science and Technology Studies (CWTS), Leiden University Leiden University s Centre for Science and Technology Studies (CWTS) provides the SNIP metrics for over 20,000 journals indexed in Scopus. These are the same SNIP measurements in Scopus but using this link they are free. SNIP (Source Normalized Impact per Paper) measures contextual citation impact by weighting citations based on the total number of citations in a subject field.

30

31 SCImago Journal & Country Rank Publicly available portal that includes the journals and country scientific indicators developed from the information contained in the Scopus database (Elsevier B.V.) Platform takes its name from the SCImago Journal Rank (SJR) indicator (PDF), developed by SCImago from the widely known algorithm Google PageRank. This indicator shows the visibility of the journals contained in the Scopus database from 1996: SJR is a prestige metric based on the idea that 'all citations are not created equal' Arguably NOT as credible as IF of 6670 journals listed, N Engl J Med is #8, Morbidity and Mortality Weekly #9, The Lancet # 14, JCI #28, JAMA #41

32 Analytic options

33 Inherent difficulties in measuring alternative forms or Impact

34 Inherent difficulties in measuring alternative forms or Impact Alt Metrics go some way towards measuring impact beyond the traditional impact factor and citation based models Altmetrics measure high level metrics associated with open rates on web sites, re-tweets, mentions in policy documentsbut they provide no insights or evidence into how that expanded coverage or exposure is influencing and changing opinion, policy and behaviour Currently a blunt instrument but it is an acknowledgement that the ultimately self referential nature of Impact Factor is not enough in and of itself

35 Editors still live and die by the impact factor (mostly ) Impact factor of any value a signal of Journal integrity (good enough to achieve an IF; hopefully eliminates rogue Journals without an adequate pipeline or peer review process) NZ Medical Journal lost its impact factor years ago when it ran into financial issues and switched from print to online (losing its ISN at the same time) lesson to us all, but NZ Med J not concerned and neither are its readers (high quality papers; very high rejection rate; impacts government; strong pipeline; strong editorial processes, editor a member of the prestigious ICJME)

36 Check your article citations are being counted correctly your IF may be too low Experience now at major Journals suggests serious errors can occur

37 1. General Recommendation Do not use journal-based metrics, such as Journal Impact Factors, as a surrogate measure of the quality of individual research articles, to assess an individual scientist s contributions, or in hiring, promotion or funding decisions. 1 of 18 recommendations offered to the general scientific research community, journal publishers, research based institutions and authors San Francisco Declaration Research Metrics

38 Measuring real impact? In health and medical research, choosing where best to publish a paper involves so much more than just the prestige (or impact factor) of the journal. In clinical and public health sciences, authors will want the right audience to read the article. When published in a surgical journal, for instance, the findings of surgical research will influence the thinking of more people, and more of those who should read the research, than when published in a more general journal, even if the latter had a higher impact factor. Similarly, public health researchers want to publish where public health policymakers and officials will read the article. A single paper in the wide-reaching Medical Journal of Australia which could change health policy and practice affecting thousands of Australians may be of greater impact than a paper in a high-impact journal that very few people read. Published in the Conversation By Warwick Anderson Professor & former Chief Executive Officer, National Health and Medical Research Council September 2013

39 The MJA achieving impact in relation to doctors health! The MJA has had a longstanding commitment to improving doctors health Over the past 5 years the MJA and MJA InSight have published widely on doctors health and the unacceptably high suicide rates of young doctors in Australia an been a vehicle for doctors to speak out Much of that has been adapted for the mainstream media

40 The MJA achieving impact in relation to doctors health! MJA and MJA InSight articles have been widely referred to or adapted for the mainstream media this has helped generate high public awareness of the issue At the AMA national conference in May 2017, Federal Health Minister Hunt announced funding and policy measures to help combat the high rates of doctor suicide MJA is making all of its published work freely available to delegates of an upcoming national conference on doctors health

41 To conclude: Defining Impact? Be clear on your purpose and define your role and goals as a Journal Are you a society journal? How does that impact your mission and purpose? Understand who you want to generate impact with and why? Do you want to: 1. Communicate with and be measured by the quality and output of your scholarly research in the local/global research community? Is a high Impact factor your major goal? 2. Improve clinical practice and health outcomes for patients? 3. Influence health system change, political decision makers and policy developments? 4. Generate public awareness and drive behavioural change in the community?

42

43 How can your Journal practically increase impact? Identify particular topics, research problems or issues that you want to focus on and develop a systematic approach or campaign: Live your mission and goals Develop and publish themed issues or sections devoted to these research problems issues or topics, e.g. Indigenous health for the MJA Actively commission quality, high profile local and global authors in this and related fields High profile authors and editors can seek to engage politicians and inform future policy Engage the mainstream media on issues of public interest and importance Use social media to drive awareness and traffic to your site If you have an adjacent medical journalism site like MJA InSight.com.au, produce adjacent and more accessible content that will get the attention of the medical and wider community

44 Measuring real impact? It s unlikely that the broader impact of scholarly medical publishing can be measured in isolation or measured empirically, there is an inevitable qualitative element to measuring these alternative ideas of impact Often broader changes in practice, policy and behaviour occur as a result of a constellation of concurrent events or activity