Maxing out on quality appraisal of your research: Avoiding common pitfalls. WITH EXAMPLES FROM THE ONGOING SETS RCT STUDY ERIC PARENT, PT, M.SC. PH.D. ASSOCIATE PROFESSOR,DEPT. OF PHYSICAL THERAPY AND ADJUNCT PROFESSOR,DEPT. OF SURGERY SCHROTH AND BSPTS THERAPIST Influencing policy and decision makers Practice guidelines, systematic reviews and individual studies influence decisions regarding funding of health care interventions Promising results from abstracts, non-randomized and poor quality studies are often dismissed by decision makers. UNIVERSITY OF ALBERTA, EDMONTON, CANADA Policy influenced by study quality Quality appraisal from systematic reviews on exercises and bracing SRS.org treatment for AIS Alternative treatments to prevent curve progression such as chiropractic medicine, physical therapy, yoga, etc. have not demonstrated any scientific value in the treatment of scoliosis. SRS e-textbook On Bracing for AIS, many orthoses have been reported effective in preventing curve progression. However, a number of patients in... these reports would not be considered appropriate for brace treatment today. These factors, along with the recognized difficulty in the objective assessment of brace wear compliance, have led to skepticism regarding the true effectiveness of brace treatment. Romano et al Cochrane lack of high quality evidence to recommend the use of SSE for AIS. better quality research needs to be conducted before the use of SSE can be recommended in clinical practice. Overall, the risk of bias in the included studies was very high. Mordecai et al Eur Spine J All studies endorsed the role of exercise therapy but shortcomings were identified Nearly all brace reports have been level of evidence III or lower
Quality appraisal from systematic reviews on Bracing Negrini et al Cochrane RCTs and prospective cohort studies should follow both the SRS and SOSORT criteria for bracing studies. Scientific evidence is in favour of bracing, but quality is very low. Checklists to appraise research quality For intervention studies RCTs and observational studies Cochrane Risk of Bias assessment http://ohg.cochrane.org/sites/ohg.cochrane.org/files/uploads/risk%of %bias%assessment%tool.pdf http://handbook.cochrane.org/ chapter_8/8_assessing_risk_of_bias_in_included_studies.htm Downs and Black, J. Epidemiol Community Health 998; 5:77-84 Armijo et al Phys Ther. 8;88:56 75. Uncontrolled studies (observational) Newcastle-Ottawa scale. http://www.ohri.ca/programs/clinical_epidemiology/oxford.asp Focus on appraisal of study quality Different than assessing the quality of reporting. More to maximize the quality of reporting Does this Intervention Help? CONSORT statement http://www.consort-statement.org/ Schulz KF, Altman DG, Moher D, for the CONSORT Group. CONSORT Statement: updated guidelines for reporting parallel group randomised trials. Ann Int Med ;5. Epub 4 March. Levels of Evidence Step (level ) Step (level ) Systematic review of randomized trials or n-of- trials Randomized trial or observational study with dramatic effect STROBE statement http://www.strobe-statement.org/ von Elm E, Altman DG, Egger M, Pocock SJ, Gøtzsche PC, Vandenbroucke JP; STROBE Initiative. The Strengthening the Reporting of Observational Studies in Epidemiology (STROBE)statement: guidelines for reporting observational studies. J Clin Epidemiol. 8 Apr;6(4):44-9. PMID: 8558 Step (level ) Non-randomized controlled cohort / follow-up study** Step 4 (level 4) Step 5 (level 5) Case-series, case-control studies, or historically controlled studies** Mechanism-based reasoning OCEBM Levels of Evidence Working Group*. "The Oxford Levels of Evidence". Oxford Centre for Evidence-Based Medicine. http://www.cebm.net/index.aspx?o=565
Differences in Quality of Exercise Research for scoliosis Summary Appraisals from Systematic reviews Only two published RCT on exercises: Wan et al + Monticone et al Monticone Romano et al Cochrane Requires appraisal of the quality of studies on exercises from lower levels of evidence. We should apply quality appraisal at the study design stage to maximize study quality at the write up stage to maximize the reporting quality Quality appraisal Quality summary : Systematic review by Mordecai et al Of 9 reviewed studies Controlled Study 7 + with historical controls Random allocation Prospective Study (only 4 controlled) Allocation Criteria described 4 ( were patient s decision) Recruitment modality described 7 Patient s Characteristics described 8 (but adequate?) Intervention described 5 (but adequate?) Blinded Assessment of outcomes NONE!!!! Confounding factors identification Confounding factors statistically controlled Many studies by authors from centres that endorse exercise therapy. Unclear indications or contraindications for treatment. None used standard functional outcome scores. Variability in radiographic and clinical parameters (only 7/ reporting Cobb angle CHG) exact method not described or nonstandardised Negrini et al. Disability and Rehabilitation 8
Quality summary : Systematic review by Mordecai et al Quality summary systematic review Negrini et al. Disab. Rehabil 8 Most report significant Cobb changes of small magnitude vs measurement error (.7 o - 6.5 o ). Compliance was poorly reported. None on long term results 7/9 /9 /9 But what is important to describe for exercise studies! 4/9 7/9 8/9 Quality summary systematic review Negrini et al. Disab. Rehabil 8 5/9 Maxing out on quality But what is important to describe for exercise studies! /9 /9 /9 4
Also: Avoid underpowered studies. If sample size is too small there is a high probability of type error. (to say there is no difference when in truth there is one). If sample size is too small the effect has to be LARGE in order for the statistics to detect it as significant. If sample size is too small it is not possible to control for confounding factors to complete subgroups analyses (REPORT for Meta-analyses) Homogenous rate of progression? Curve increase was. / month before the curve acceleration phase. Curve increase was.9 / month following the curve acceleration phase (.,.8, and.6 in the low, moderate, and rapid groups.) Rapid curve progression lasts - years beyond the initiation of the curve acceleration phase Begins around peak height velocity Ends with Risser stage 4. Sanders et al JBJS 89 A jan 7 All patients in the rapid subgroup had the most common curve types (major thoracic or double major with large thoracic). Rate of progression vs age and curve severity Plan a study where you can detect a large enough effect to obtain significance with your sample Solid line Expected change in controls may differ: Larger if mostly patients from rapid progression during curve acceleration phase Smaller if mostly patients before or after the curve acceleration phase Depend on curve size after maturity. Dashed lines Plan studies of subjects with similar risk of progression. less variability = easier to detect treatment effects 5
Common risk of bias in research on conservative treatments for scoliosis Bias A bias is a systematic error, or deviation from the truth, in results or inferences. can operate in either direction: lead to underestimation or overestimation of the true intervention effect. can vary in magnitude: some are small and some are substantial an apparent finding may be entirely due to bias Usually impossible to know to what extent biases have affected the results of a particular study. http://bmg.cochrane.org/assessing-risk-bias-included-studies Selection Bias Selection Bias Systematic differences between baseline characteristics of the groups that are compared. Randomization if successful prevents selection bias in allocating interventions to participants. AN IMPORTANT RISK OF BIAS IN EXERCISE / BRACE studies AFFECTS GENERALIZABILITY TO REPRESENTATIVE SUBJECTS. Describe a good recruitment strategy: GOOD: representative of target population, consecutively enrolled from your clinic over a representative period or randomly invited to participate. Explain which kind of center. Demonstrate enrolled patients are similar to those excluded. Selection criteria reflect consensus (treatment is indicate. BAD: Cherry picking, not explaining source of recruitment. Convenience 6
SRS Standardization criteria for AIS brace studies ) selection. Optimal inclusion criteria consist of age years and older when the orthosis is prescribed, Risser -, primary curve magnitude 5-4, no prior treatment, if female, either premenarchal or less than year postmenarchal. AT RISK OF PROGRESSION Need to include documentation of: age when first diagnosed (juvenile vs. adolescent). menarchal status, curve patterns, curve rotation (Nash-Moe or the Perdriolle method).. Allocation sequence generation GOOD: The investigators describe a random component in the sequence generation process such as: Referring to a random number table; Using a computer random number generator; Coin tossing; Shuffling cards or envelopes; Throwing dice; Drawing of lots; eg. www.random.org (sequence generator) e.g www.randomization.com (randomly permuted blocks) BAD: odd or even date of birth; rule based on date (or day) of admission; rule based on hospital or clinic record number; by judgement of the clinician; by preference of the participant; based on the results of a laboratory test or a series of tests; by availability of the intervention.. Allocation sequence concealment Cobb Angle improvement at 6 months Biased by baseline curve severity Participants and investigators could not foresee assignment because an adequate method used to conceal allocation: GOOD: Central allocation (including telephone, web-based and pharmacycontrolled randomization); Sequentially numbered drug containers of identical appearance; Sequentially numbered, opaque, sealed envelopes. BAD: Using an open random allocation schedule (e.g. a list of random numbers); Assignment envelopes were used without appropriate safeguards; Alternation or rotation; Date of birth; Case record number; Any other explicitly unconcealed procedure. 4,5 4,5,5,5,5 Schroth Standard care Due to curve serverity at baseline Selection bias prevents detecting the good treatment effect 7
Cobb Angle improvement at 6 months NOT Biased by baseline curve severity Cobb Angle improvement at 6 months Biased by baseline curve severity 6 4,5 5 4,5 4 Schroth Standard care Due to curve serverity at baseline Randomization protects against selection bias and allows detecting the good treatment effect,5,5,5 Schroth With adequate sample size can imbalanced can be controlled statistically Standard care Due to curve serverity at baseline Selection bias prevents detecting the good treatment effect Performance Bias. Blinding of participants and personnel Systematic differences between groups in the care that is provided, or in exposure to factors other than the interventions of interest. AVOID differences between groups!!! In SETS Attention differs between groups. Blinding of study participants and personnel may reduce the risk that knowledge of which intervention was received, rather than the intervention itself, affects outcomes. Blinding can also ensure that the compared groups receive a similar amount of attention, ancillary treatment and diagnostic investigations. GOOD: No blinding or incomplete blinding, but the outcome is not likely to be influenced by lack of blinding; Blinding of participants and key study personnel ensured, and unlikely that the blinding could have been broken. SETS STUDY Participants and personnel not blinded ALMOST IMPOSSIBLE to blind PARTICIPANTS IMPOSSIBLE to blind THERAPISTS Avoid Single or Double blind terms. SPECIFY! 8
Detection bias 4. BLINDING OF OUTCOME ASSESSMENT Systematic differences between groups in how outcomes are determined. Blinding of outcome assessors may reduce the risk that knowledge of which intervention was received, rather than the intervention itself, affects outcome measurement. Blinding is especially important for assessment of subjective outcomes GOOD: No blinding of outcome assessment, but the outcome measurement is not likely to be influenced by lack of blinding; Blinding of outcome assessment ensured, and unlikely that the blinding could have been broken. (ask evaluators to guess) If you can t BLIND maybe provide evidence of no bias. SETS STUDY: YES BUT The patients know. Effect on questionnaire??? Effect on surface topography? (Objective but try harder?) Effect on radiographs?( try harder?) Blinding in exercise studies? Cobb Angle improvement at 6 months biased by lack of blinding NONE used blinded examiners Patients therapists statisticians. 8 7 6 5 4 Due to lack of blinding Schroth Standard care Performance and detection bias lead to overestimating the good treatment effect Negrini et al. Disability and Rehabilitation 8 9
Attrition Bias Systematic differences between groups in withdrawals from a study/ compliance with prescribed treatment. Withdrawals from the study lead to incomplete outcome data. Exclusions refer to situations in which some participants are omitted from reports of analyses, despite outcome data being available to the trialists. Attrition refers to situations in which outcome data are not available. GOOD: 5. Incomplete Outcome Data No missing outcome data; Reasons for missing outcome data unlikely to be related to true outcome; Missing outcome data balanced in numbers across intervention groups, with similar reasons for missing data across groups; The plausible difference in means among missing outcomes not enough to have a clinically relevant impact on the observed effect size; Missing data have been imputed using appropriate methods. SETS STUDY Intent-to-treat analysis Drop-outs reported, low and balanced Report missing data even in retrospective studies Cobb Angle improvement at 6 months biased by attrition Reporting Bias,5,5,5,5 Per protocol Attrition bias leads to overestimating the good treatment effect Schroth Standard care Schroth Standard care If dropout in Schroth were because unhappy, Poor outcomes. If dropout in Standard were for balanced reason,5,5,5 Intent-to-treat Systematic differences between reported and unreported findings. Within a published report those planned analyses with statistically significant differences between intervention groups are more likely to be reported than planned analyses with non-significant differences. (Selective reporting)
GOOD: 6. Selective reporting The study protocol is available and all of the study s pre-specified (primary and secondary) outcomes of interest have been reported in the pre-specified way; The study protocol is not available but it is clear that the published reports include all expected outcomes, including those that were prespecified (convincing text of this nature may be uncommon). Trial Registration A clinical trials registry, is an official platform and catalog for registering a clinical trial. The goal of a clinical trials registry is to provide increased transparency and access to clinical trials, made available to the public. Trials are registered by who is running the study. ClinicalTrials.gov includes both interventional and observational studies. SETS is registered: NCT698 Protocol published: Schreiber S et al J Physiother. 4 6 4 Other Bias Carry-over in cross-over trials bias Recruitment bias e.g. SETS volunteers may be keeners! Contamination, whereby the experimental and control interventions get mixed E.g. some SETS Standard Care patients planned trips to get Schroth therapy as inpatients! Summary Pick the right study design (RCT and controlled studies most influential) Recruit large enough samples + homogenous samples (Recommendations) Assess meaningful outcomes and plan interventions for sufficiently large effects. Avoid common biases Selection, Randomize, Blinding, Attrition/compliance, Standardize interventions Thank you! Interested in Graduate Studies, please contact : Eparent@ualberta.ca Meet to discuss opportunities.