376 Journal of Traditional Chinese Medicine, December 2011; 31(4): 376-381 Evidence-based Study Assessing the Quality of the First Batch of Evidence-Based Clinical Practice Guidelines in Traditional Chinese Medicine YU WEN-ya 1, XU Jian-long 2, SHI Nan-nan 1, WANG Li-ying 1, HAN Xue-jie 1, WANG Yong-yan 1, Lv Ai-ping 1 Objective: To assess the quality of the first batch of Chinese evidence-based clinical practice guidelines (CPGs) in Traditional Chinese Medicine (TCM) using the Appraisal of Guidelines for Research Evaluation (AGREE) instrument. Methods: Evidence-based CPGs in TCM supported by the World Health Organization Western Pacific Regional Office (WHO/WPRO) whose development was organized by the China Academy of Chinese Medical Sciences were identified manually retrieved. CPGs were assessed using the AGREE instrument, the data in each CPG were analyzed in terms of the six domains in the AGREE instrument: scope purpose, stakeholder involvement, rigor of development, clarity presentation, applicability, editorial independence. Results: Twenty-eight CPGs were identified, of which 26 were included in the study. The AGREE instrument rated the 26 CPGs in terms of the six domains. The assessment results showed the following average scores: for editorial independence, 84.16%; for rigor of development, 80.95%; for scope purpose, 79.96%; for clarity presentation, 70.88%; for stakeholder involvement, 61.28%; for applicability, the average score was only 27.09%. In summary, nine CPGs were rated as strongly recommended, six as recommended with provision or alternation, 11 as unsure. Conclusion: Most of the first batch of Chinese evidence-based CPGs in TCM had significant shortcomings in applicability. It is suggested that special attention be paid to enhancing the quality of applicability when developing Keywords: clinical practice guideline; Traditional Chinese Medicine; evidence-based medicine; AGREE instrument; quality assessment Clinical practice guidelines (CPGs) are systematically developed to assist practitioners patients in making decisions about the appropriate health care in specific clinical circumstances. 1 CPGs in Traditional Chinese Medicine (TCM) are basic technical stards are helpful in clinical practice. CPGs in TCM have been increasingly used in medical decision-making, guiding junior TCM practitioners biomedical doctors willing to practice TCM. However, the potential benefits of the guidelines are only as good as the quality of the guidelines themselves: 2 only good guidelines have the potential to improve clinical practice. 3 Consensus-based CPGs in TCM have been developed for many diseases; however, their quality varies greatly, some CPGs do not achieve an acceptable stard. 4 In 2006, under the support of the World Health Organization Western Pacific Regional Office (WHO/WPRO), multidiscipline panels were convened by the China Academy of Chinese Medical Sciences (CACMS) to develop the first batch of evidence-based CPGs in TCM in China, though no quality evaluation has thus far been conducted. Before applying evidence-based CPGs in TCM, the first step is to determine their quality. This paper is aimed at assessing the quality of METHODS The first batch of evidence-based CPGs in TCM were published in TCM journals in China; 5,6 they were identified manually retrieved. The evidence-based CPGs in TCM were assessed using the Appraisal of Guidelines for Research Evaluation (AGREE) instrument. The AGREE instrument was developed by a group of researchers from 13 nations in 1998 has been used in some countries to assess CPGs. 7-11 The AGREE instrument consists of 23 key items organized in six domains. The six domains are scope purpose, stakeholder involvement, rigor of development, clarity presentation, applicability, editorial 1. Institute of Basic Research in Clinical Medicine, China Academy of Chinese Medical Sciences, Beijing 100700, China; 2. Xiyuan Hospital of China Academy of Chinese Medical Sciences, Beijing 100700, China Correspondence to: Prof. LÜ Ai-ping, Email: lap64067611@126. com This study was financially supported by projects from the China Academy of Chinese Medical Sciences (No.Z0135), the State Administration of Traditional Chinese Medicine (No. ZYYS-2008), the National Science Foundation of China (No. 30825047).
Journal of Traditional Chinese Medicine, December 2011; 31(4): 376-381 377 independence. Each domain is intended to capture a separate dimension of guideline quality. Taking each appraisal criterion into account, an overall assessment of CPGs should be made by at least two appraisers. The overall assessment contains a series of options: Strongly recommended, Recommended with provisions or alterations, Not recommended, Unsure. 12 The data of each CPG were gathered then analyzed in terms of the six domains of the AGREE instrument. After completing the AGREE assessment, two appraisers reached consensus on the overall recommendation for each CPG. RESULTS Twenty-eight EB CPGs in TCM were identified, of which 26 met the inclusion criteria. The 26 diseases were as follows: diabetes, chronic obstructive pulmonary disease, IgA nephropathy, acquired immunodeficiency syndrome, obesity, common cold, hypertension, coronary heart disease (angina pectoris), age-related macular degeneration, aplastic anemia, rheumatoid arthritis, chronic pelvitis, chronic gastritis, chronic hepatitis B, cerebral infarction, migraine, chronic prostatitis, Barton fracture, cervical spondylotic radiculopathy, insomnia, atopic dermatitis, pediatric pneumonia, vascular dementia, psoriasis vulgaris, depression, primary osteoporosis. The analysis results of the quality of evidence-based CPGs in TCM are shown in Table 1-1, 1-2, 1-3. Table 1-1. Six evidenced-based CPGs in TCM were evaluated using the AGREE instrument (all published in Chinese) Title Supported AGREE (six domains) by Nian Ling Xiang Guan Xing Huang Ban Bian Xing CPGs convening organization Scope purpose Stakeholder involvement Rigor of development Clarity presentation Editorial independence WHO/WPRO CACMS 77.8% 77.8% 66.7% 63.5% 51.9% 88.9% Zai Sheng Zhang Ai Xing Pin Xue Lei Feng Shi Xing Guan Jie Yan Man Xing Pen Qiang Yan Zhong Shi Jian Man Xing Wei Yan Nao Geng Si WHO/WPRO CACMS 87.9% 83.3% 87.3% 75% 33.3% 94% WHO/WPRO CACMS 83.3% 45.8% 91.7% 77.4% 0.0% 100% WHO/WPRO CACMS 81.5% 63.0% 77.8% 65.1% 25.9% 77.8% WHO/WPRO CACMS 92% 88% 86% 81% 69% 100% Notes: Nian Ling Xiang Guan Xing Huang Ban Bian Xing Zhi Nan: Clinical Practice Guideline of Traditional Chinese Medicine for Age-Related Macular Degeneration (Chin); 6 Zai Sheng Zhang Ai Xing Pin Xue Zhi Nan: Clinical Practice Guideline of Traditional Chinese Medicine for Aplastic Anemia (Chin); 6 Lei Feng Shi Xing Guan Jie Yan Zhong Yi Lin Zhi Nan: Clinical Practice Guideline of Traditional Chinese Medicine for Rheumatoid Arthritis (Chin); 5 Man Xing Pen Qiang Yan Zhi Nan: Clinical Practice Guideline of Traditional Chinese Medicine for Chronic Pelvitis (Chin); 6 Man Xing Wei Yan Zhi Nan: Clinical Practice Guideline of Traditional Chinese Medicine for Chronic Gastritis (Chin); 5 Nao Geng Si Zhi Nan: Clinical Practice Guideline of Traditional Chinese Medicine for Cerebral Infarction (Chin). 5
378 Journal of Traditional Chinese Medicine, December 2011; 31(4): 376-381 Table 1-2. Ten evidenced-based CPGs in TCM were evaluated using the AGREE instrument (all published in Chinese) Title 2 Xing Tang Niao Bing Zhong Yi Lin Chuang Shi Jian Man Xing Zu Sai Xing Fei Ji Bing IgA Shen Bing Huo De Xing Mian Yi Que Xian Zong He Zheng 5 Dan Chun Xing Fei Pang Zhong Shi Jian Gan Mao Zhong Shi Jian Gao Xue Ya Bing Man Xing Wen Ding Xing Xin Jiao Tong Zhong Shi Jian Pian Tou Tong Man Xing Yi Xing Gan Yan Supported by CPGs convening organization Scope purpose Stakeholder involvement AGREE (six domains) Rigor of development Clarity presentation Editorial independence WHO/WPRO CACMS 84.1% 56.4% 81.3% 67.8% 0.0% 82.7% WHO/WPRO CACMS 66.6% 56.2% 60.4% 64.3% 0.0% 70.8% WHO/WPRO CACMS 66.7% 44.4% 75.0% 63.5% 0.0% 95.0% WHO/WPRO CACMS 63.3% 47.2% 63.8% 55.5% 0.0% 83.3% WHO/WPRO CACMS 74.1% 50.0% 88.9% 69.8% 66.7% 83.3% WHO/WPRO CACMS 81.5% 69.4% 83.3% 77.8% 51.9% 77.8% WHO/WPRO CACMS 93% 67% 79% 83% 52% 94% WHO/WPRO CACMS 83.33% 68.75% 64.58% 60.71% 55.56% 87.50% WHO/WPRO CACMS 94.4% 66.0% 88.0% 70.8% 66.6% 100% WHO/WPRO CACMS 92.59% 66.67% 87.18% 94.67% 25.93% 100% Notes: 2 Xing Tang Niao Bing Zhi Nan: Clinical Practice Guideline of Traditional Chinese Medicine for Type 2 Diabetes (Chin); 5 Man Xing Zu Sai Xing Fei Ji Bing Zhi Nan: Clinical Practice Guideline of Traditional Chinese Medicine for Chronic Obstructive Pulmonary Disease (Chin); 5 IgA Shen Bing Zhi Nan: Clinical Practice Guideline of Traditional Chinese Medicine for IgA Nephropathy (Chin)); 5 Huo De Xing Mian Yi Que Xian Zong He Zheng: Clinical Practice Guideline of Traditional Chinese Medicine for Acquired Immunodeficiency Syndrome (Chin); 5 Dan Chun Xing Fei Pang Zhong Yi Lin Zhi Nan: Clinical Practice Guideline of Traditional Chinese Medicine for Obesity (Chin); 5 Gan Mao Zhi Nan: Clinical Practice Guideline of Traditional Chinese Medicine for the Common Cold (Chin); 5 Gao Xue Ya Bing Zhi Nan: Clinical Practice Guideline of Traditional Chinese Medicine for Hypertension (Chin); 5 Man Xing Wen Ding Xing Xin Jiao Tong Chuang;Shi Jian Zhi Nan: Clinical Practice Guideline of Traditional Chinese Medicine for Coronary Heart Disease (angina pectoris) (Chin); 5 Pian Tou Tong Zhi Nan: Clinical Practice Guideline of Traditional Chinese Medicine for Migraine (Chin). 5 Man Xing Yi Xing Gan Yan Zhi Nan: Clinical Practice Guideline of Traditional Chinese Medicine for Chronic Hepatitis B (Chin). 5
Journal of Traditional Chinese Medicine, December 2011; 31(4): 376-381 379 Table 1-3. Six evidenced-based CPGs in TCM were evaluated using the AGREE instrument (all published in Chinese) Title Rao Gu Yuan Duan Gu Zhe Shen Jing Gen Xing Jing Zhui Bing Zhong Yi Lin Chuang Shi Jian Shi Mian Zheng Te Ying Xing Pi Yan Zhong Yi Lin Chuang Shi Jian Xiao Er Fei Yan Xue Guan Xing Chi Dai Zhong Shi Jian Zhi Nan 5 Xun Chang Xing Yin Xie Bing Yi Yu Zheng Yuan Fa Xing Gu Zhi Shu Song Zheng Supported by CPGs convening organization Scope purpose Stakeholder involvement AGREE (six domains) Rigor of Clarity development presentation Editorial independence WHO/WPRO CACMS 85.2% 69.4% 91.7% 87.3% 59.3% 100% WHO/WPRO CACMS 88.9% 63.9% 88.9% 73.0% 45.80% 100% WHO/WPRO CACMS 66.7% 50.0% 79.4% 52.8% 0.0% 77.8% WHO/WPRO CACMS 83.3% 60.4% 77.1% 79.7% 25.0% 87.5% WHO/WPRO CACMS 77.8% 61.1% 91.67% 74.6% 18.5% 88.9% WHO/WPRO CACMS 92% 83% 92% 83% 57% 83% Man Xing Qian WHO/WPRO CACMS 66.6% 55.5% 80.5% 55.9% 0.0% 83.3% Lie Xian Yan Notes: Rao Gu Yuan Duan Gu Zhe Zhi Nan: Clinical Practice Guideline of Traditional Chinese Medicine for Barton Fracture (Chin; 6 Shen Jing Gen Xing Jing Zhui Bing Zhi Nan: Clinical Practice Guideline of Traditional Chinese Medicine for Cervical Spondylotic Radiculopathy (Chin); 6 Shi Mian Zheng Zhi Nan: Clinical Practice Guideline of Traditional Chinese Medicine for Insomnia (Chin); 5 Te Ying Xing Pi Yan Zhi Nan: Clinical Practice Guideline of Traditional Chinese Medicine for Atopic Dermatitis (Chin); 5 Xiao Er Fei Yan Chuang Shi Jian Zhi Nan: Clinical Practice Guideline of Traditional Chinese Medicine for Pediatric Pneumonia (Chin); 6 Xue Guan Xing Chi Dai Zhong Yi Lin Zhi; Nan: Clinical Practice Guideline of Traditional Chinese Medicine for Vascular Dementia (Chin); 5 Xun Chang Xing Yin Xie Bing Zhi Nan: Clinical Practice Guideline of Traditional Chinese Medicine for Psoriasis Vulgaris (Chin); 6 Yi Yu Zheng Zhi Nan: Clinical Practice Guideline of Traditional Chinese Medicine for Depression (Chin); 5 Yuan Fa Xing Gu Zhi Shu Song Zheng Zhi Nan: Clinical Practice Guideline of Traditional Chinese Medicine for Primary Osteoporosis (Chin); 5 Man Xing Qian Lie Xian Yan Zhi Nan: Clinical Practice Guideline of Traditional Chinese Medicine for Chronic Prostatitis (Chin). 6
380 Journal of Traditional Chinese Medicine, December 2011; 31(4): 376-381 Scope Purpose The majority of CPGs presented their scope purpose well, scoring over 60% in this domain. Among the 26 CPGs, five scored 60% 70%, seven scored 70% 80%, nine 80% 90%, five over 90%. The average domain score was 79.96%. Stakeholder Involvement Target users were not defined well in most guidelines, the majority of CPG development groups did not involve the patient s perspective. Among the 26 CPGs, 12 scored under 60%, 10 scored between 60% 70%, one between 70% 80%, three between 80% 90%; the average score was 61.28%. Rigor of Development Nearly all the CPGs described the search methods for evidence in detail, the levels of evidence grading of recommendation were reported clearly: in this way, the link between recommendation supporting evidence was explicit. However, the balance between benefits side effects or risks was rarely considered. All CPGs underwent external expert review before being published, the CPG updating procedure was considered. Overall, the average score for this domain was 80.95%; four CPGs scored over 90%, 13 between 80% 90%, five between 70% 80%, four between 60% 70%. Clarity Presentation The key recommendations of most CPGs were easy to identify, but the clarity of the recommendations needs to be improved. Some CPGs indicated no tools for application. Three CPGs scored under 30%, 11 between 60% 70%, seven between 70% 80%, four between 80% 90%, one over 90%; the average domain score was 70.88%. Few CPGs clearly defined potential cost implications or provided any detail of barriers posed by other organizations in applying the CPGs, though they did present performance-monitoring indicators. The applicability score was the lowest one among the six domains: 11 CPGs scored zero, 15 between 60% 70%; the average score was 27.09%. Editorial Independence Most CPGs defined editorial independence or conflicts of interest among the guideline authors. Four CPGs scored between 70% 80%, 13 between 80% 90%, nine over 90%; the average score was 84.16%. Overall Assessment On the whole, nine of twenty-six CPGs were rated as strongly recommended, six CPGs were rated as recommended with provisions or alterations, 11 CPGs were rated as unsure. DISCUSSION The expert committee the multidisciplinary panels were convened by CACMS in 2006. Under the guidance of the expert committee, the multidisciplinary panels developed the first batch of evidence-based CPGs in TCM in China from 2006 to 2011. Before developing the CPGs, the expert committee stipulated that all members of the multidisciplinary panels be well trained, it provided a common structure techniques relating to CPGs for the multidisciplinary panels. The development procedure adhered to that of a Scottish Intercollegiate guideline network, assessment of the quality levels 13 recommendation grading 14 of the published evidence was conducted. Where the published evidence was inadequate, expert consensus was reached. After drafting of a CPG was completed, it was reviewed by external experts; then, three or four appraisers without personal conflicts of interest were selected to evaluate the CPG independently using the AGREE instrument. Some countries have developed their own CPG appraisal instruments. A systematic review of 24 different appraisal tools some studies has shown that the AGREE instrument is an acceptable stard for guideline evaluation. 16-18 The expert committee recommended that the AGREE instrument be used as a tool to assess the quality of the first batch of The AGREE assessment showed that the quality of the evidence-based CPGs in TCM was higher than that of some other CPGs in the literature reviewed. 7,8;10,11 The average scores for five domains scope purpose, stakeholder involvement, rigor of development, clarity presentation, editorial independence were all over 50%, though the average score for applicability was only 27.09%. The reason for the low average score in applicability was that 11 guidelines scored zero in this domain. contains items relating to organizational barriers, cost/resource implications for recommendations, key review criteria for monitoring /or audit purposes. Nearly half of the CPGs gave no consideration of organizational barriers in guideline implementation; instead, they supplied monitoring criteria to assess the guideline s impact. This may partly be because the expense of traditional herbal drugs is low, decoctions consisting of a number of herbs are popularly used in clinical practice. It is difficult to give a clear definition about potential cost implications or details of the administrative impact of applying CPGs. The present study has some limitations. This paper focused only on evidence-based CPGs in TCM, it did not include other type of guidelines. For example, the consensus-based guidelines that constitute a high proportion of CPGs in TCM 4 were not included. In addition, the sample size of the CPGs was relatively small. Despite these limitations, this study clearly
Journal of Traditional Chinese Medicine, December 2011; 31(4): 376-381 381 showed the overall quality of the first batch of evidence-based CPGs in TCM in China. In conclusion, the majority of evidence-based CPGs in TCM have significant shortcomings in applicability. This suggests that special attention should be paid to enhancing the quality of applicability when developing REFERENCES 1. Committee to advise the public health service on clinical practice guidelines, Institute of medicine. In: Field MJ, Lohr KN, editors. Clinical practice guidelines: directions for a new program. Washington (DC): National Academy Press; 1990. 2. Melissa C. Brouwers, Michelle E. Kho, George P. Browman, Jako S. Burgers, Francoise Cluzeau, Gene Feder, et al. Development of the AGREE, part 1:performance, usefulness areas for improveement. CMAJ 2010; 182: 1045-1052. 3. Carneiro AV. Methodological appraisal of guidelines. The AGREE instrument. Rev Port Cardiol 2004; 23: 447-456. 4. Yuwen Y, Han XJ, Shi NN, Wang LY, Xie YM, Lu AP. Development of consensus method in establishing clinical practice guideline with traditional Chinese medicine approaches. SH.J.TCM (Chin) 2011; 45: 15-19. 5. Cao HX, Wang YY. Evidence-based guideline of clinical practice in Chinese medicine internal China: CPTCM (Chin); 2011. 6. Cao HX, Wang YY. Evidence-based guideline of clinical practice in Chinese medicine specific disease. China: CPTCM (Chin); 2011. 7. Gallardo CR, Rigau D, Irfan A, Ferrer A, CaylàJ. A, Bonfi ll X, et al. Quality of tuberculosis guidelines: urgent need for improvement. Int J Tuberc Lung Dis 2010; 14: 1045-1051. 8. Kinnunen-Amoroso M, Pasternack I, Mattila S, Parantainen A. Evaluation of the practice guidelines of Finnish institute of occupational health with AGREE instrument. Ind Health 2009; 47: 689-693. 9. Loveday BP, Srinivasa S, Vather R, Mittal A, Petrov MS, Phillips RJ, et al. High quality variable quality of guidelines for acute pancreatitis: a systematic review. Am J Gastroenterol 2010; 105: 1466-1476. 10. Alonso-Coello P, Irfan A, Solà I, Gich I, Delgado-Noguera M, Rigau D, et al. The quality of clinical practice guidelines over the last two decades: a systematic review of guideline appraisal studies. Qual Saf Health Care 2010; 19: 1-7. 11. Gorman SK, Chung MH, Slavik RS, Zed PJ, Wilbur W, Dhingra VK. A critical appraisal of the quality of critical care pharmacotherapy clinical practice guidelines their strength of recommendations. Intensive Care Med 2010; 36: 1798-1801. 12. AGREE instrument. (Accessed December 5, 2010 at Http: //www.agreecollaboration.org). 13. Liu JP. The composition of evidence body of traditional medicine recommendations for its evidence grading. Chin J Integr Tradit West Med (Chin) 2007; 27: 1061-1065. 14. GRADE Working Group. Grading quality of evidence strength of recommendations. BMJ 2004; 328: 1490-1494. 15. Vlayen J, Aertgeerts B, Hannes K, Sermeus W, Ramaekers D. A systematic review of appraisal tools for clinical practice guidelines: multiple similarities one common deficit. Int J Qual Health Care 2005; 17: 235-242. 16. AGREE Collaboration. Development validation of an international appraisal instrument for assessing the quality of clinical practice guidelines: the AGREE project. Qual Saf Health Care 2003; 12: 18-23. 17. MacDermid JC, Brooks D, Solway S, Switzer -Mclntyre S, Brosseau L, Graham ID. Reliability validity of the AGREE instrument used by physical therapists in assessment of clinical practice guidelines. BMC Health Serv Res 2005; 5: 18. (Received June 20, 2011)