Assessing risk of bias

Transcription

1 Assessing risk of bias Norwegian Research School for Global Health Atle Fretheim Research Director, Norwegian Institute of Public Health Professor II, Uiniversity of Oslo

2 Goal for the day We all have an understanding of the term «risk of bias», and are able to apply it when we assess research studies

3 Systematic review Systematic retrieval, critical assessment and summary of research on a specific research question (often on effects of an intervention) Critical assessment entails an assessment of whether the results are trustworthy (internal validity, «bias») If misleading study findings are included in a systematic review, the findings of the systematic review will likely be misleading too

4 Misleading studies misleading SR RR (95 % CI) for death Study 1 Study 2 Study 3 Study 4 Study 5 Overall effect 0.5 1

5 Mesleading studies misleading SR RR (95 % CI) for death Study 1 Study 2 Study 3 Study 4 Study 5 Overal effect «True» result «True» result Misleading result Misleading result Misleading result 0.5 1

6 Mesleading studies misleading SR RR (95 % CI) for death Study 1 Study 2 Study 3 Study 4 Study 5 Overall effect «True» result «True» result Bad study Bad study Bad study 0.5 1

7 What can lead to misleading results? Antibiotics Still in pain after 3 days? Child with middle ear infection No antibiotics Still in pain after 3 days?

8 «Bias» - not always easy to define Skewedness Prejudice Something inherently wrong

9 «Bias» Systematic errors that lead to erroneous (non- «true») results

10 Risk of Bias (RoB) Directly related to internal validity the likelihood that the results reflect «the truth» Low RoB: We think the results are likely «true» High RoB: We think the results may be «untrue» Not related to precision an imprecise result can be «true» (is caused by random errors, not systematic errors) Not related to external validity (applicability, transferability) a study finding can be «true» even if it s non-applicable to a different setting!

11 Difference between RoB and quality Examples: Quality can imply more than RoB (e.g. ethical aspects are important, but not for RoB) Unreasonable to classify a solid non-randomized study as «low quality» especially when randomisation is impossible (will still likely have high RoB, though!) Not always possible to «blind» patients doesn t mean that the study is of «low quality»? (not necessarily of high RoB either!)

12 Risk of Bias (RoB)-assessment An approach to evaluating risk of systematic errors in a study, or of a study finding The question is: What is it about this study that gives us reason to doubt the truthfulness of the findings? In other words: We are looking for possible sources of «bias» in a study. Several check lists out there we recommend Cochrane s «Risk of Bias Tool»

13 Assess what s written in the paper, or what the researchers actually did? Poor reporting often makes it difficult to assess the risk of bias It s not what the researchers say they did, or don t say they did, that we re after we re after what they actually did But still: Our assessment will largely depend on the information we re able to access (the paper, and possibly direct contact with the authors)

14 Summarised so far We wish to assess the risk that systematic errors may have affected the study results risk of bias (internal validity) The assessment depends on the study s design and the how it was conducted (which does not necessarily correspond to «study quality»!) Important to realise right away: Some degree of judgement is unavoidable when assessing risk of bias you need to think and draw your conclusions as best you can (unfortunately )

15 The Cochrane RoB-tool emphasises six 1. Sequence generation domains* 2. Allocation concealment 3. Blinding of participants, personell and outcome assessors 4. Incomplete follow up of outcome measures 5. Selective reporting 6. Other risks of bias * Developed especially for RoB-assessment of RCTs, but the same approach can, in prinicple, be applied on all types of effectiveness studies.

16 Why these domains? A combination of logic/theory and empirical findings E.g. it s been shown that in studies were allocation wasn t conecealed (i.e. it was possible to predict which group the next participant would be allocated to), the results were different* * Pildal et al. Impact of allocation concealment on conclusions drawn from meta-analyses of randomized trials. Int J Epidemiol 2007.

17 Example of a problem: Testing of serum treatment for pneumonia ( )

18 Example of a problem: Testing of serum treatment for pneumonia ( )

19 Example of a problem: Testing of serum treatment for pneumonia ( )

20 Example of a problem: Testing of serum treatment for pneumonia ( ) The researchers found that There were far fewer who were given than were not given the treatment (the numbers should ve been equal!) The average age was lower among those who received the treatment (should ve been equal!)

21 The Cochrane RoB-tool emphasises six 1. Sequence generation domains* 2. Allocation concealment 3. Blinding of participtans, personell and outcome assessors 4. Incomplete follow up of outcome measures 5. Selective reporting 6. Other risks of bias * Developed especially for RoB-assessment of RCTs, but the same approach can, in prinicple, be applied on all types of effectiveness studies.

22 Trial to evaluate use of antibiotics Sequence generation Allocation concealment Blinding Incomplete follow up? Selective reporting? Amoxicillin to 161 No improvement by 3rd day: children with acute otitis media Placebo to 158 Other risks of bias? No improvement by 3rd day: 22 Source: Tähtinen et al, N Engl J Med 2011; 364:

23 Trial to evaluate use of antibiotics Sequence generation Allocation concealment Amoxicillin to 161 No improvement by 3rd day: children with acute otitis media Placebo to 158 No improvement by 3rd day: 22 Source: Tähtinen et al, N Engl J Med 2011; 364:

24 Sequence generation Sequence generation: Method used to decide the order of allocation (e.g. coin tossing, premade randomisation list, every other etc.)

25 Trial to evaluate use of antibiotics Sequence generation Amoxicillin to 161 No improvement by 3rd day: children with acute otitis media Placebo to 158 No improvement by 3rd day: 22 Source: Tähtinen et al, N Engl J Med 2011; 364:

26 Sequence generation and concealed allocation Sequence generation: Method used to decide the order of allocation (e.g. coin tossing, premade randomisation liste, every other etc.) Concealed allocation: None of those involved know which group the next participant will end up in (until the participant is included in the study)

27 Trial to evaluate use of antibiotics Sequence generation Allocation concealment Amoxicillin to 161 No improvement by 3rd day: children with acute otitis media Placebo to 158 No improvement by 3rd day: 22 Source: Tähtinen et al, N Engl J Med 2011; 364:

28 Why is the sequence generation and concealed allocation important? To ensure comparable groups, i.e. prevent allocation bias Sequence generation and allocation concealment are inter-linked Non-random allocation makes it difficult to achieve concealed allocation Unconcealed allocation can undermine randomisation

29 Trial to evaluate use of antibiotics Sequence generation Allocation concealment Blinding Amoxicillin to 161 No improvement by 3rd day: children with acute otitis media Placebo to 158 No improvement by 3rd day: 22 Source: Tähtinen et al, N Engl J Med 2011; 364:

30 Concealed allocation vs. blinding Concealed allocation: No one knows which group the next participant will be allocated to (before the participant is included in the study) Blinding: Neither personnel, participants or those who assess outcomes are aware of to which group the participants are allocated to (also after inclusion in the trial)

31 Blinding Was the group affiliation of the participants kept secret in an adequate way during the study, for the participants? for the personnel? for those who assessed outcomes? TAKE NOTICE! Blinding can vary from one outcome to another within the same study (both whether blinding was done, and if lack of blinding is a RoB)

32 Why is blinding important? If the participants know which treatment they re receiving, that can in itself affect the result (e.g. due to expectations) Knowing which treatment has been given can also affect the assessment of outcomes (by the participant him-/herself, or others who assess the treatment result) Blinding is likely more important for subjective outcomes, than objective

33 Incomplete follow up of outcomes Were missing data managed in a satisfactory way (participants who didn t meet or dropped out)? If data were not included in the analyses (exclusion of participants), was a good reason given? «Intention to treat» (ITT) is a key concept Optimal ITT, outcomes are assessed on all participants, and all are included in all analyses TAKE NOTICE! The degree of incomplete follow up may vary from one outcome to another

34 Trial to evaluate use of antibiotics Sequence generation Allocation concealment Blinding Incomplete follow up? Amoxicillin to 161 No improvement by 3rd day: children with acute otitis media Placebo to 158 No improvement by 3rd day: 22 Source: Tähtinen et al, N Engl J Med 2011; 364:

35 Why is incomplete follow up a potential problem? The fate of participants who dropped out or who were excluded during the course of the study, may have an impact on the results if they had been included in the analysis (e.g. did they drop out because they fell ill?)

36 Number randomised Risk among observed Observed data Hypothetical extreme risks among missing Missing data Complete data Risk ratio Trial A Intervention % 225/450 80% 40/50 265/500 Control % 225/450 20% 10/50 235/

37 Number randomised Risk among observed Observed data Hypothetical extreme risks among missing Missing data Complete data Risk ratio Trial A Intervention % 225/450 80% 40/50 265/500 Control % 225/450 20% 10/50 235/ Trial B Intervention % 45/450 80% 40/50 85/500 Control % 45/450 20% 10/50 55/

38 Selective reporting Are there no signs of selective reporting of results?

39 Trial to evaluate use of antibiotics Sequence generation Allocation concealment Blinding Incomplete follow up? Selective reporting? Amoxicilling to 161 No improvement by 3rd day: children with acute otitis media Placebo to 158 No improvement by 3rd day: 22 Source: Tähtinen et al, N Engl J Med 2011; 364:

40 Why is it important to assess the risk of selective reporting? It s been shown that researchers often choose to report only some of their findings typically «positive» results This leads to bias in systematic reviews, since usually only results that are reported are included I.e. we risk being fooled if we only see the «positive» results

41 Other risks of bias Are there no other reasons to suspect or believe that there are other issues that may entail a high risk of bias?

42 Trial to evaluate use of antibiotics Sequence generation Allocation concealment Blinding Incomplete follow up? Selective reporting? Amoxicillin to 161 No improvement by 3rd day: children with acute otitis media Placebo to 158 Other risks of bias? No improvement by 3rd day: 22 Source: Tähtinen et al, N Engl J Med 2011; 364:

43 Why are other risk of bias important? There are plenty of thinkable sources of bias in a study Therefore, we need to think about possible sources of bias in the study we re assessing Try to understand how the study was actually conducted, in practice! Was the study ended earlier than planned? Cheating?

44 RoB-assessmen entails two steps 1. Figure out how the study was conducted (what was done?) 2. Decide for yourselves whether this entails low, high or unclear risk of bias (current classification in the Cochrane RoB-tool)

45 RoB a key step in doing a systematic review Relevant studies Critical assessment of each of them (RoB)

46

47

48 As mentioned: The Cochrane RoB-tool emphasises six domains* 1. Sequence generation 2. Allocation concealment 3. Blinding of participants, personnel and outcome assessors 4. Incomplete follow up of outcome measures 5. Selective reporting 6. Other risks of bias * Developed especially for RoB-assessment of RCTs, but the same approach can, in prinicple, be applied on all types of effectiveness studies.

49

50

51 RoB Entry Judgement Description Satisfactory sequence generation? YES (Low) Unclear NO (High) In order to convince the participating clinics that it was a fair decision who got the intervention and who did not, the head of the clinics were present when we tossed a coin to decide allocation.

52 RoB Entry Judgement Description Satisfactory sequence generation? YES (Low) Unclear NO (High) Using a random computergenerated sequence, which was kept sealed until the end of the study, pharmacy staff prepared syringes that were filled with saline or with influenza vaccine... Clinic staff could not distinguish vaccine from placebo.

53 RoB Entry Judgement Description Satisfactory sequence generation? Concealed allocation? Blinding? YES (Low) Unclear NO (High) Using a random computergenerated sequence, which was kept sealed until the end of the study, pharmacy staff prepared syringes that were filled with saline or with influenza vaccine... Clinic staff could not distinguish vaccine from placebo.

54 RoB Entry Judgement Description Concealed allocation? YES (Low) Unclear NO (High) Used a random computer generated sequence

55 Overall RoB-assessment per study (or per outcome) We need to make an overall assessment: Is the risk of bias for this, in this study high or low (or unclear)? If we assess the RoB as low (or high) across all domains, the answer is given If not a bit more complicated

56 How to make a RoB-assesment across several studies? RoB of bias may be different across studies Several options: For your systematic review, you could choose only to include studies where the RoB is low Include all studies, but discuss problems with RoB Conduct several analyses (sensitivity analyses)

57 The Cochrane Handbook suggests several criteria for assessing RoB But you still need to make your own judgement (and use your own brain)

58 Was the allocation sequence adequately generated? YES, when The investigators describe a random component in the sequence generation process such as: Referring to a random number table; Using a computer random number generator; Coin tossing; Shuffling cards or envelopes; Throwing dice; Drawing of lots; Minimization*. *Minimization may be implemented without a random element, and this is considered to be equivalent to being random.

59 Was the allocation sequence adequately generated? NO, when The investigators describe a non-random component in the sequence generation process, for example: Sequence generated by odd or even date of birth; Sequence generated by some rule based on date (or day) of admission; Sequence generated by some rule based on hospital or clinic record number..

60 Was the allocation sequence adequately generated? UNCLEAR, when Insufficient information about the sequence generation process to permit judgement of Yes or No.

61 Was allocation adequately concealed? YES, when Participants and investigators enrolling participants could not foresee assignment because one of the following, or an equivalent method, was used to conceal allocation: Central allocation (including telephone, webbased, and pharmacy-controlled, randomization); Sequentially numbered drug containers of identical appearance; Sequentially numbered, opaque, sealed envelopes.

62 Was allocation adequately concealed? NO Participants or investigators enrolling participants could possibly foresee assignments and thus introduce selection bias, such as allocation based on: Using an open random allocation schedule (e.g. a list of random numbers); Assignment envelopes were used without appropriate safeguards (e.g. if envelopes were unsealed or non-opaque or not sequentially numbered); Alternation or rotation; Date of birth; Case record number; Any other explicitly unconcealed procedure.

63 Was allocation adequately concealed? UNCLEAR, when Insufficient information to permit judgement of Yes or No. This is usually the case if the method of concealment is not described or not described in sufficient detail to allow a definite judgement for example if the use of assignment envelopes is described, but it remains unclear whether envelopes were sequentially numbered, opaque and sealed.

64 Was knowledge of the allocated interventions adequately prevented (blinding)? YES, when Any one of the following: No blinding, but the review authors judge that the outcome and the outcome measurement are not likely to be influenced by lack of blinding; Blinding of participants and key study personnel ensured, and unlikely that the blinding could have been broken; Either participants or some key study personnel were not blinded, but outcome assessment was blinded and the non-blinding of others unlikely to introduce bias.

65 Was knowledge of the allocated interventions adequately prevented (blinding)? No, when Any one of the following: No blinding or incomplete blinding, and the outcome or outcome measurement is likely to be influenced by lack of blinding; Blinding of key study participants and personnel attempted, but likely that the blinding could have been broken; Either participants or some key study personnel were not blinded, and the non-blinding of others likely to introduce bias.

66 Was knowledge of the allocated interventions adequately prevented (blinding)? UNCLEAR Any one of the following: Insufficient information to permit judgement of Yes or No ; The study did not address this outcome.

67 Were incomplete outcome data adequately addressed? YES, when Any one of the following: No missing outcome data; Reasons for missing outcome data unlikely to be related to true outcome; Missing outcome data balanced in numbers across intervention groups, with similar reasons for missing data across groups;

68 Cont d. For dichotomous outcome data, the proportion of missing outcomes compared with observed event risk not enough to have a clinically relevant impact on the intervention effect estimate; For continuous outcome data, plausible effect size (difference in means or standardized difference in means) among missing outcomes not enough to have a clinically relevant impact on observed effect size; Missing data have been imputed using appropriate methods.

69 Were incomplete outcome data adequately addressed? NO, when Any one of the following: Reason for missing outcome data likely to be related to true outcome, with either imbalance in numbers or reasons for missing data across intervention groups; For dichotomous outcome data, the proportion of missing outcomes compared with observed event risk enough to induce clinically relevant bias in intervention effect estimate;

70 Cont d. For continuous outcome data, plausible effect size (difference in means or standardized difference in means) among missing outcomes enough to induce clinically relevant bias in observed effect size; As-treated analysis done with substantial departure of the intervention received from that assigned at randomization; Potentially inappropriate application of simple imputation.

71 Were incomplete outcome data adequately addressed? UNCLEAR, when Any one of the following: Insufficient reporting of attrition/exclusions to permit judgement of Yes or No (e.g. number randomized not stated, no reasons for missing data provided); The study did not address this outcome.

72 Are reports of the study free of suggestion of selective outcome reporting? YES, when Any of the following: The study protocol is available and all of the study s pre-specified (primary and secondary) outcomes that are of interest in the review have been reported in the pre-specified way; The study protocol is not available but it is clear that the published reports include all expected outcomes, including those that were pre-specified (convincing text of this nature may be uncommon).

73 Are reports of the study free of suggestion of selective outcome reporting? NO, when Any one of the following: Not all of the study s pre-specified primary outcomes have been reported; One or more primary outcomes is reported using measurements, analysis methods or subsets of the data (e.g. subscales) that were not prespecified;

74 Cont d. One or more reported primary outcomes were not pre-specified (unless clear justification for their reporting is provided, such as an unexpected adverse effect); One or more outcomes of interest in the review are reported incompletely so that they cannot be entered in a meta-analysis; The study report fails to include results for a key outcome that would be expected to have been reported for such a study.

75 Are reports of the study free of suggestion of selective outcome reporting? UNCLEAR, when Insufficient information to permit judgement of Yes or No. It is likely that the majority of studies will fall into this category.

76 Was the study apparently free of other problems that could put it at a risk of bias? YES, when The study appears to be free of other sources of bias.

77 Was the study apparently free of other problems that could put it at a risk of bias? NO, when There is at least one important risk of bias. For example, the study: Had a potential source of bias related to the specific study design used; or Stopped early due to some data-dependent process (including a formal-stopping rule); or Had extreme baseline imbalance; or Has been claimed to have been fraudulent; or Had some other problem.

78 Was the study apparently free of other problems that could put it at a risk of bias? UNCLEAR There may be a risk of bias, but there is either: Insufficient information to assess whether an important risk of bias exists; or Insufficient rationale or evidence that an identified problem will introduce bias.