Title: Selection effects may account for better outcomes of the German Disease Management Program for type 2 diabetes

Author's response to reviews Title: Selection effects may account for better outcomes of the German Disease Management Program for type 2 diabetes Authors: Ingmar Schäfer (in.schaefer@uke.uni-hamburg.de) Claudia Küver (c.kuever@uke.uni-hamburg.de) Benjamin Gedrose (b.gedrose@uke.uni-hamburg.de) Eike Christin von Leitner (eleitner@uke.de) András Treszl (a.treszl@uke.uni-hamburg.de) Karl Wegscheider (k.wegscheider@uke.uni-hamburg.de) Hendrik van den Bussche (bussche@uke.uni-hamburg.de) Hanna Kaduszkiewicz (kaduszki@uke.uni-hamburg.de) Version: 2 Date: 9 November 2010 Author's response to reviews: see over

Institut für Allgemeinmedizin Prof. Dr. Hendrik van den Bussche Direktor Zentrum für Psychosoziale Medizin Martinistraße 52 20246 Hamburg Telefon: (040) 7410-52400 Telefax: (040) 7410-53681 bussche@uke.uni-hamburg.de Universitätsklinikum Hamburg-Eppendorf Martinistraße 52 20246 Hamburg Institut für Allgemeinmedizin The Editor BMC Health Services Research Dr.rer.nat. Eike-Christin von Leitner Telefon: (040) 7410-58543 Telefax: (040) 7410-53681 eleitner@uke.uni-hamburg.de November, 9 th 2010 Manuscript resubmission Paper 1089816661405464 TITLE: Selection effects may account for better outcomes of the German Disease Management Program for type 2 diabetes Dear Editors, Thank you very much for the report of the reviewers and for the opportunity to resubmit a revised version of this paper. We highly appreciate the constructive criticism and have revised the paper carefully. We hope to have met all the suggestions satisfactorily. On the following pages you will find answers to each comment. To facilitate reading, the original comments of the reviewers were copied into this letter. Reviewer: Rolf Holle P.1, Abstract: The longitudinal character of the study is mentioned in the Methods section, but not in the Results section of the abstract. Is that result not important or is it due to the authors lack of confidence in the result? We did not mention the results of the longitudinal analyses of GhB and blood pressure in the abstract due to our lack of confidence in outcome results after having found the selection bias. This is stated on page 12 of the submission: "Our conclusion therefore is that the existing selection bias of more active and motivated DMP patients with a lower risk for diabetic complications makes it impossible to decide if the DMP is responsible for the differences in the course of BP between DMP and usual-care patients."

P.2, Background: please check the writing style which in my opinion does not fit in some places ( can be a thankless task, DMP is a cure-all solution ). We received positive and negative feedback regarding our writing style concerning the two examples given. We changed "DMP is a cure-all solution" to "DMP is a highly effective intervention". However, we kept the first sentence including "can be a thankless task", because it points to a problem many researchers may face in the evaluation of grand scale interventions. P.4, Methods and Fig.1: The widely differing exclusion rates (46% vs. 70%) between both groups are one of the main problems of this paper. a) It would be important to have more specific information about this. b) In addition, do the authors have information about DMP participation rates between practices and did they consider using this information in the analysis? c) Please also report the size of cluster effects for different variables. a) For clarification we amended our discussion regarding the widely differing participation rates between DMP and usual-care patients in the "strengths and weaknesses" chapter (pp.12-13) by the underlined passages: "DMP and usual-care patients had different response rates, i.e. 70% of DMP and 46% of usual-care patients agreed to participate in the study. In part, this difference can be explained by the inferior patient documentation in usual-care patients including more invalid postal addresses for contact. The difference in response rates may also be a part of the DMP selection bias: In both groups the more active and motivated patients may also be more often interested in study participation. For this reason the number of usualcare patients with good self-management may be larger in our study than it is de facto. We thus may have even underestimated the selection bias." b) We did not collect data regarding the total number of DMP and usual-care patients in each surgery and we therefore cannot compute DMP participation rates between practices. c) We provide the intraclass correlation coefficients (ICC) as requested: Variable (at baseline) ICC Age 0,08 Gender 0,06 Duration of diabetes 0,09 Co-morbidity: depression 0,01 Smoking behaviour 0,07 GHb value 0,06 Systolic blood pressure 0,03 Oral medication 0,06 Urinary glucose self-monitoring 0,38 Blood glucose self-monitoring 0,13 Prescription of diabetes education before DMP start 0,30 GP-rated motivation of the patients 0,27 We based our reporting on the Strobe Checklist for case-control studies and other observational studies (http://www.strobe-statement.org/), which do not include intraclass correlation coefficients. ICCs are rarely reported. All multivariate analyses conducted in our

paper account for the intraclass correlations reported here. In our opinion mentioning them in our paper could confuse readers. Therefore we would not like to add this information to the manuscript. P.6, Missing data and Table 1&2: a) Table 1 may not be necessary, the information could be reported in the text. b) More important: is BMI among the variables excluded from analysis because of too many missing data? Results on BMI are not reported but should be. c) Would it be possible to identify a subset of participating centres which have nearly complete data, do a sensitivity analysis in this subset and report it in the manuscript? a) We reported the imputation of missing data according to the requirements of Sterne JAC, White IR, Carlin JB, Spratt M, Royston P, Kenward MG et al: Multiple imputation for missing data in epidemiological and clinical research: potential and pitfalls. BMJ 2009;338:b2393. We would like to keep table 1, because it allows a more differentiated description than reporting the missing cases in the text. In addition the second reviewer wished even more detailed information on missing cases. We therefore extended the table by differentiating missing cases between usual-care and DMP patients. b) The BMI was excluded from analyses because of too many missing values, i.e. the proportion of missing cases was more than 22% (60% in usual care patients and 5% in DMP patients) at baseline. We did not accept variables with a total of 21% and more missing cases based on Unnebrink K, Windeler J: Intention-to-treat: methods for dealing with missing values in clinical trials of progressively deteriorating diseases. Stat Med 2001;20:3931-46. We judge the variable "BMI" as not valid and would like to keep it excluded from the analyses. c) In all participating surgeries only 30% of usual-care patients (=76 cases) have a complete data set. There are no surgeries with complete or nearly complete data sets. It seems likely that GPs know more about patients with good self-management than about patients with low motivation and activity. If we had performed regression analyses on complete cases only we might have biased our results. Therefore we based this publication on results from imputed data and reported results from both, original and imputed data. P.7, Results: Does the mean age refer to baseline or follow-up? Please report the observation period separately for both groups. A table of baseline characteristics should be provided Referring to the comments of the referee we changed the first paragraph of the results chapter as follows: "The analysis is based on 586 DMP patients and 250 usual-care patients. Usualcare patients were significantly older at baseline (67.4 ± 12.5 years vs. 66.0 ± 10.3 years) and had a shorter duration of illness than DMP patients (5.2 ± 5.5 years vs. 6.8 ± 7.2 years). 52.6% of the study participants were male. No gender differences between DMP patients and usual-care patients were found. The average observation period between t 0 and t 1 was 34 ± 17 months for DMP patients and 40 ± 5 months for usual-care patients, respectively. Differences in baseline characteristics between DMP and usual-care patients are shown in Table 3." We included a table with baseline characteristics (cf. Table 3) and amended the methods, statistical analyses chapter by "Differences in baseline characteristics between DMP and usual-care patients were analysed by bivariate t-tests and χ 2 -tests."

P.8, Outcome quality and Table 4: in Table 4 the expression difference for DMP patients may be misunderstood. Please also add adjustment variables in footnote a) Table 4 is now named table 5. By computing odds ratios we described differences in the course of GHb and systolic blood pressure between t 0 and t 1 for DMP patients related to the reference group of usual-care patients. We changed the expression in the table to DMP versus usual-care patients. b) We added the following text to the table footnote "Adjusted for length of observation period between t 0 and t 1, age, gender, duration of diabetes, depression, medication (regarding GHb only), smoking status (regarding BP only) and diabetes education activity of the GP." P.9 and table 5: a) Either report odds ratios (and CI) in table 5 or delete the table. b) Second paragraph on p.9 is interpretation and should be in the Discussion section. a) We included odds ratios and 95% confidence intervals into table 5, which now is table 6. b) As suggested, we moved the second paragraph on p.9 into the discussion chapter. P.11 top: a) why did the authors not analyze antihypertensive medication, which may be more frequent in the DMP group and thereby explain the results for this outcome? b) As long as these data are not provided the lengthy discussion in the upper half of p.11 seems speculative. a) We did not collect medication data. The main goal of our study was to determine possible selection mechanisms in the process of enrolment into the DMP and to study their impact on DMP effectiveness and the assessment of DMP outcome measures. It was not our aim to study the effectiveness of the different DMP components - and initiating or changing medication is one of the DMP components. b) The last paragraph on p.11 discusses the consequences of having found a selection bias for interpreting the results in outcome measures. We agree with the reviewer that antihypertensive medication given in the DMP might account for the better development of blood pressure values, but this does not explain the differences in blood pressure at baseline. If the antihypertensive medication had been given more often to future DMP patients before they were enrolled into the program, this might endorse our hypothesis that actively treated patients were more likely to be enrolled into the DMP. Medication could therefore be one of the "factors that are unmeasured" that are mentioned in our manuscript. This underlines the statement that it is not possible to statistically control for the total selection effect. We regard the paragraph on p. 11 essential for understanding our study and decided to keep it.

Strength and Weaknesses: a) The longitudinal design of the study with a retrospective baseline may not be optimal for the main aim concerning selection effects, this should be discussed. b) In addition, do the authors have any idea whether the selection of participating physicians could generate bias? a) After the DMP had been introduced into the health care system on a nationwide basis, unfortunately it was not possible to perform much better studies. This objective is introduced at the very beginning of the paper: Health services research can be a thankless task. Policy makers depend on scientific evidence, but their decisions sometimes restrain researchers from generating this evidence in a valid way. This is the case with the nationwide disease management program (DMP) for type 2 diabetes, which was introduced in Germany in 2003. Further issues concerning the case-control design are being discussed on p.11: "It is important to decide if the bias found in our study mainly results from discontinuation of DMP enrolment (drop out) [17] or from GPs selection mechanisms in clinical reality. Case-control studies (and all other studies recruiting patients after the intervention started) may suffer from a selection bias occurring when patients systematically drop out of the intervention group before data collection (e.g. because of low motivation or severe illness). We examined this possibility in our study: We found 21 usual-care patients who temporarily had been enrolled in the DMP before the study started. It was not possible to find out the number of former DMP patients who had died or changed the GP, but we suppose that their number was small. To estimate the selection bias due to drop out we performed a 'quasi'-intention to treat analysis by treating usual-care patients as DMP patients if they had ever been enrolled in the DMP in the past. The results of this analysis were only marginally different from the results presented in this paper. Patients still had a better chance for enrolment if they had a lower risk for diabetic complications (i.e. odds ratio 1.9 for lower systolic BP), were more frequently treated with oral medication (2.6) and had higher self-activity rates (i.e. 3.8 for urine and 1.8 for blood glucose monitoring and 2.9 for having attended a diabetes education program) and higher GP-rated motivation (4.6). The outcome analyses showed the same findings as stated in the results chapter of this paper. We therefore suppose that there is a considerable difference between DMP and usual-care patients at baseline irrespective of a selection bias due to drop out." b) We are aware that physicians participating in our study could be more interested in research than not participating physicians, but from experience with these physicians during the course of the study it is unlikely that this has caused any serious bias. The GPs whose data were analysed highly differed regarding their appreciation of the DMP and their compliance with DMP guidelines irrespective of their interest in research. This reflects the variety of opinions found in German GPs. For this reason, we are quite confident that the selection of GPs did not affect our results.

Reviewer: James Ralston Major Compulsory Revisions 1. a) Temper conclusions according to the potential for bias originating from participation in the study vs. not participating. b)although the authors make a reasonable argument that the lower differential study participation rate among those not in the DMP group may actually underestimate the real differences between the DMP and nondmp group, the study cannot determine the direction of the potential bias. a) We tempered our conclusions on page 15: "We conclude that self-active and motivated patients with a lower risk for diabetic complications seem to be more likely to be enrolled in the German DMP. The selection mechanisms of the DMP in part reflect the legal regulations. Policy makers explicitly defined that only active patients with potential benefit should be enrolled. This question touches aspects of appropriateness of care, equity and accessibility to healthcare. The major clinical implication of our results is that the disease management program probably does not reach a considerable amount of higher risk patients. " We also adjusted the conclusions in the abstract: At baseline, future DMP patients had a lower risk for diabetes complications, were treated more intensively and were more active and motivated in managing their disease than usual-care patients. This finding a) points to the problem that the German DMP may not reach the higher risk patients and b) selection bias may impair the assessment of differences in outcome quality between enrolled and usual-care patients. Suggestions for dealing with this bias in evaluation studies are being made. b) We agree with the reviewer that we cannot prove the direction of the selection bias. However, although the participation rates differ between DMP and usual-care patients, we found highly significant differences in baseline values between these groups. As discussed in our manuscript, these results clearly show the direction that self-active and motivated patients with a lower risk for diabetic complications have a better chance to be enrolled into the program. We included approximately 50% of eligible usual-care patients. In order to reverse our conclusions the other 50% of usual-care patients (those, who did not participate in the study) would need to show much better values in self-management and motivation than the DMP patients in our study. Furthermore, the non participating usual-care patients (but not the non-participating DMP patients) would need to show a very strong positive correlation between non-participation in our study, good self-management and good motivation for lifestyle changes, respectively. This seems rather unlikely. We are therefore quite confident that our study can determine the direction of the selection bias. 2. Discuss limitations of comorbidity measures. The authors rightly point out that those with the highest healthcare needs, or comorbidity, are more likely not being reached by DMP. The measures of comorbidity, though, are limited to HTN and depression. There are no overall comorbidity measures, such as Charlson or ACGs. The study also does not include other burdensome conditions associated with diabetes (such as CHF, CAD, renal failure). The demands of these other conditions often compete for the attention of patients and providers against needs to address standard DM outcomes like A1c, BP, LDL. Since DMP likely targets mainly standard DM outcomes, the presence of such advanced comorbidity may further keep patients from entering the DMP program and

represent a lost opportunity for the kind of proactive care and follow-up that the DMP group is intended to receive. We agree with the referee that further comorbidity may also keep from being enrolled into the DMP. For discussion we included a paragraph into the Strengths and Weaknesses chapter on page 12: "Except for depression and hypertension, we did not collect standardized data on the patients' comorbidity. We showed that patients with a higher risk for diabetic complications are less likely to be in the DMP group. Analogously the presence of burdensome comorbidity might also keep patients from being enrolled into the program and may therefore also contribute to the selection bias. This hypothesis should be analysed in further research." 3. Discussion section includes a description of additional analyses looking at potential selection bias due to drop out from DMP program. These are results and should be in the results section with a description of the methods in the methods section. We did as suggested. We described the methods in the statistical analyses section on page 7: "To estimate the selection bias due to drop out we performed 'quasi'-intention to treat analyses for the assessment of selection bias and outcome quality by treating usual-care patients as DMP patients if they had ever been enrolled in the DMP in the past. These analyses used the same statistical modelling as described above." The results were moved to a results subchapter called "Quasi-intention-to-treat-analysis" on page 9: "We found 21 usual-care patients who temporarily had been enrolled in the DMP before the study started. If we count these patients as DMP patients, the results from our analysis are only marginally different from the results presented in this paper (cf. Table 7)." The discussion of this passage on page 10-11 was shortened: "It is important to decide if the bias found in our study mainly results from discontinuation of DMP enrolment (drop out) [18] or from the GPs selection mechanisms in clinical reality. Case-control studies (and all other studies recruiting patients after the intervention started) may suffer from a selection bias occurring when patients systematically drop out of the intervention group before data collection (e.g. because of low motivation or severe illness). We examined this possibility in our study by a 'quasi'-intention to treat analysis counting usual-care patients as DMP patients if they had ever been enrolled in the DMP in the past. Both, assessment of selection bias and analysis of outcome quality showed similar results to our main analyses. We therefore suppose that there is a considerable difference between DMP and usual-care patients at baseline irrespective of a selection bias due to drop out." Minor Essential Revisions 1. Missing data: Measurement bias for DM outcomes may need to be accounted for in evaluations between DMP and nondmp groups. Although there is some good description of data missingness for the study, I couldn t find where authors described differential missinginess of DM outcomes, such as A1c BP LDL measures, between the DMP and nondmp group. We followed the suggestion and differentiated table 1 into total missings, missings of usualcare patients and missings of DMP patients.

2. Odds Ratios. a) Since this is a policy relevant paper, OR may be difficult for general readers to understand. The outcomes measured are common so the OR do not likely approximate RR. So, if you stick with OR, less aware readers seeing an OR above 4 may incorrectly infer the magnitude of difference as a RR. b) In the results, I recommend that the authors state the meaning of at least one of OR for one of the outcomes. For the other OR presented in the text include OR as a label and the 95% CI. a) We introduced a paragraph that explains how odds ratios convert into risk ratios (citing Zhang J, Yu KF: What's the Relative Risk? A Method of Correcting the Odds Ratio in Cohort Studies of Common Outcomes. JAMA 1998;280:1690-1691) and give two examples on p. 10: "We compared the chances for enrolment by odds ratios, which are standard measures for these analyses. Odds ratios are less intuitionally understood than risk ratios because they are based on differences in odds, not in likelihood as risk ratios (e.g. a number on a die has a 1/5 odds, but a 1/6 likelihood). An odds ratio of 4.55 for GP-rated high motivation vs. low motivation of the patients means that a patient with high motivation has 4.55-times higher odds for enrolment than a person with low motivation. If exposure variables have a low prevalence (i.e. 5% and below) odds ratios and risk ratios are comparable. As the factors we identified in this study are common (prevalence > 10%), the odds ratios (OR) presented in Table 4 do not approximate risk ratios (RR). It is possible to convert OR into RR by accounting for the frequency of each factor [17]. In doing this the 4.55 odds ratio (with a total prevalence of high motivation of 20.54% in the sample) translates to a 2.63 risk ratio. This means that a highly motivated patient has 2.63- times higher likelihood for enrolment than a person having a low motivation." 3. Describe what the outcome or outcomes were for the power calculation. It appears that power calculations may be for A1c and BP, but it is not clear in the text. It is also not clear what the meaningful difference was for which the power was calculated. In our manuscript the following details about the power calculation can be found on p.7/8: "we defined 1% in GHb [15] and 10 mmhg in systolic BP [16] as clinically relevant differences" and "Statistical power (1-β) for outcome differences in both GHb and systolic BP was > 99% considering the existing sample size." These statements should answer the referee's questions. 4. Typos and English The paper has a few typos and needs some changes in language. These include, a) The use of the word development in the text and tables when referring to outcomes needs to be changed. E.g., page 8 development of systolic BP. I believe the authors want say something like systolic BP was different or systolic BP was lower in the DMP vs. nondmp group. b) Page 11, I believe recourse is meant to be the overall response or participation for the study, though it is not clear. If the authors are referring to an overall 60% response or participation rate, it is less meaningful than describing the individual participation rates in the DMP and nondmp groups (ie 70 and 47%).

a) We replaced the word "development" by the word "course". This should clarify that we refer to the delta between baseline and follow-up (now page 9). b) We replaced the word "recourse" by "overall participation rate". The individual rates for DMP and usual-care patients are mentioned in the text. We present the overall rate for comparison with other studies that don't report the participation rate for the two groups separately (now page 12). Yours sincerely Dr. Eike-Christin von Leitner