Are free text records a possible source of detection bias in Clinical Practice Research Datalink studies? A case control study For peer review only

Transcription

1 Are free text records a possible source of detection bias in Clinical Practice Research Datalink studies? A case control study Journal: Manuscript ID bmjopen-0-0 Article Type: Research Date Submitted by the Author: -Feb-0 Complete List of Authors: Price, Sarah; University of Exeter Medical School Stapley, Sal; University of Exeter Medical School Shephard, Elizabeth; University of Exeter Medical School Barraclough, Kevin; Hoyland House Hamilton, Willie; University of Exeter Medical School <b>primary Subject Heading</b>: Epidemiology Secondary Subject Heading: General practice / Family practice Keywords: PRIMARY CARE, STATISTICS & RESEARCH METHODS, ONCOLOGY : first published as 0./bmjopen-0-0 on May 0. Downloaded from on April 0 by guest. Protected by copyright. -

2 Page of Are free text records a possible source of detection bias in Clinical Practice Research Datalink studies? A case control study Sarah J. Price*: Medical School, University of Exeter, College House, St Luke s Campus, Exeter, UK, EX LU, tel: +--, S.J.Price@exeter.ac.uk Sal A. Stapley: Medical School, University of Exeter, College House, St Luke s Campus, Exeter, UK, EX LU, S.A.Stapley@exeter.ac.uk Elizabeth Shephard: Medical School, University of Exeter, College House, St Luke s Campus, Exeter, UK, EX LU, E.A.Shephard@exeter.ac.uk Kevin Barraclough:Hoyland House, Gyde Road, Painswick, Gloucestershire, UK, GL RD, UK, k.barraclough@btinternet.com William T. Hamilton: Medical School, University of Exeter, College House, St Luke s Campus, Exeter, UK, EX LU, W.Hamilton@exeter.ac.uk *Correspondence to: Dr Sarah J. Price, Medical School, University of Exeter, College House, St Luke s Campus, Exeter, UK, EX LU, tel: +--, S.J.Price@exeter.ac.uk Key words: Cancer, Disease modelling, General practice, Methodology, Research design in epidemiology Word count: : first published as 0./bmjopen-0-0 on May 0. Downloaded from on April 0 by guest. Protected by copyright. -

3 Page of ABSTRACT Objectives: To estimate under-recording and bias in studies of Clinical Practice Research Datalink (CPRD) data that restrict analyses to Read codes, omitting anything recorded as text. Design: Matched case control study. Setting: Patients contributing data to the CPRD. Participants:, bladder, and, pancreatic, cancer cases diagnosed between January 000 and December 00, matched on age, sex and GP practice to up to controls (bladder: n=,; pancreas: n=,). The analysis period was the year before cancer diagnosis. Primary and secondary outcome measures: Frequency of haematuria, jaundice and abdominal pain, grouped by recording style: Read code or text-only. The association between recording style and case/control status (chi-squared test). For each feature, the odds ratio (conditional logistic regression) and positive predictive value (PPV; Bayes' Theorem) for cancer, before and after addition of text-only records. Results: Text-only records identified,/0, (%) individual attendances for the features. Text-only recording was more strongly associated with controls than with cases for haematuria (0/=% vs /,=%) in bladder cancer (chi-squared test, p<0.00), and for jaundice (/=% vs /,=0%, p<0.000) and abdominal pain (/,=% vs /,=%, p<0.00) in pancreatic cancer. Adding text-only records reduced PPVs of haematuria for bladder cancer from.0% (%CI:..%) to.% (..%) and of jaundice for pancreatic cancer from.% (..%) to.% (..%). Adding text-only records did not alter the PPV of abdominal pain for bladder cancer (Read codes: 0.%, 0. 0.% vs Read codes plus text-only records: 0.%, 0. 0.%) or for pancreatic cancer (0.%, 0. 0.% vs 0.%, %). : first published as 0./bmjopen-0-0 on May 0. Downloaded from on April 0 by guest. Protected by copyright. -

4 Page of Conclusions: Omission of text records from CPRD studies introduces detection bias that inflates outcome measures for recognised alarm symptoms. This potentially reinforces clinicians views of the known importance of these symptoms, marginalising the significance of low-risk but not no-risk symptoms. ARTICLE SUMMARY Strengths and limitations of this study Set in the CPRD, in which electronic medical records are recorded prospectively, eliminating recall bias. Large study size. Comprehensive identification of coded records of cancer symptoms. Search criteria for identifying text records of cancer symptoms will have missed US spellings and spelling errors. Study did not identify whether text records arose through the recording practices of a small number of GP practices. FUNDING STATEMENT Sarah J Price is funded by a University of Exeter PhD studentship. COMPETING INTERESTS The author(s) declare that they have no competing interests. INTRODUCTION The Clinical Practice Research Datalink (CPRD) is a UK-based research service providing anonymised copies of primary care records. It is the largest of its kind and a representative sample of the UK population. Though created for pharmacological research, the database is now used extensively in epidemiological studies.[]the outputs of such research are used as evidence to underpin national guidelines; for example, estimates of cancer risk in symptomatic patients presenting to primary care.[, ] Quality standards for the CPRD devised for pharmaco-epidemiology research have not been amended to accommodate areas such as symptom-based research. : first published as 0./bmjopen-0-0 on May 0. Downloaded from on April 0 by guest. Protected by copyright. -

5 Page of One potential, but neglected, concern for those using CPRD data arises from the way clinical events are recorded. Most CPRD practices use ViSion software (ViSion INPS, London, UK), in which GPs must choose a Read code to begin a record, after which a comments box opens. Here, GPs can type freely and are not limited to elaborating on the code. While codes are fully and routinely available to researchers, text records are not. Furthermore, a moratorium on collection of CPRD text data was introduced in 0. Text recorded before 0 can be accessed, but this is rarely done because the methods are complex, expensive and limited. Recording style may generate detection bias researchers being oblivious to anything that is only recorded as an inaccessible text comment (henceforth called hidden text ).[] In a previous CPRD case control study, we identified that of patients with non-visible haematuria, the records of attendance for (%) were made solely in hidden text. The proportional loss of records in the hidden text was similar in cases and controls. Therefore, the effect of the detection bias was limited to underestimation of the frequency of non-visible haematuria.[] This paper further investigates the potential consequences for research of having two recording methods codes and hidden text. Specifically, we sought evidence for the disproportionate use of hidden text to record symptoms/signs presented by cases and controls, as this will result in biased estimates of outcome measures such as odds ratios (ORs) and positive predictive values (PPVs). METHOD This study extended two CPRD studies conducted by our group that identified the features of bladder and pancreatic cancer in primary care.[, ] This was deliberate, in order to investigate whether detection bias affects the many cancer diagnostic studies conducted since 000, all restricting their analysis to coded CPRD data.[,, -] Setting, patients and period of study : first published as 0./bmjopen-0-0 on May 0. Downloaded from on April 0 by guest. Protected by copyright. -

6 Page of CPRD data were used in a matched case control design. Cases ( 0 years old) had a bladder/pancreatic cancer-specific code recorded between January 000 and December 00 inclusive. Each case was matched with up to five controls on age, sex and GP practice. The diagnosis date for each case, taken as their earliest recorded cancer-specific code, determined the endpoint of the -year analysis period for each case control set. Full inclusion and exclusion criteria for participants are available on request. Variables We examined whether records of attendance for three features of cancer identified in the original studies were vulnerable to detection bias through loss of records in the hidden text. The features were chosen as being representative of alarm or 'low-risk but not no-risk' features of cancer; namely, visible haematuria (an alarm symptom for bladder cancer), jaundice (an alarm feature of pancreatic cancer) and abdominal pain (a low-risk symptom common to both cancers). Identifying attendance for visible haematuria, jaundice or abdominal pain from coded records The medical records of all participants were searched for coded evidence of attendance for visible haematuria, jaundice or abdominal pain. The original bladder cancer study s code list for visible haematuria was re-used.[] Patients whose medical record contained none of the codes were assumed not to have experienced this symptom. This process was repeated using the jaundice code list created in the pancreatic cancer study.[] For abdominal pain, which is common to both cancers, the original studies had used slightly different code lists. Codes for indigestion and dyspepsia were included in the pancreatic, but not the bladder, cancer study. For the study reported here, a unified abdominal pain code list was constructed (thus, numbers for abdominal pain differ slightly from those originally reported). Full code lists are available on request. : first published as 0./bmjopen-0-0 on May 0. Downloaded from on April 0 by guest. Protected by copyright. -

7 Page of Identifying attendance related to visible haematuria, jaundice or abdominal pain from text records The CPRD were asked to search the hidden text of all participants for the following terms (optimised to catch variations in capitalisation): abdominal pain: 'bdominal pai', 'bdo pai', 'ain in abdo', 'pigastric pai' jaundice: 'aundice', 'cterus', 'cteric' haematuria: 'aematuria','lood in urine' The CPRD anonymised each extract and provided text strings of the search term plus three words either side to aid interpretation. Generating variables from hidden text extracts Hidden text extracts were converted to binary variables (symptom presence/absence) using an algorithm that identified descriptors suggesting symptom absence (e.g. no visible haematuria ) or uncertainty (e.g. if any ). Extracts flagged as uncertain were classified manually in consultation with WH and KB, both practising GPs. If an extract s meaning could not be agreed, or it was confirmed that the extract was uninterpretable, the symptom was deemed to be absent. The algorithm s validity was assessed by comparing its output with that of a gold standard. The gold standard was created using the panel of consensus method, in which WH and KB independently rated a random sample of text extracts about visible haematuria.[] The level of agreement between the algorithm and the gold standard was high (chance-corrected weighted Kappa=0., % CI 0..). Full details of gold standard construction are available on request. Identifying text-only recording For each patient, date matching of coded and text records identified attendances for each feature that were recorded solely in hidden text. Binary variables were created to identify attendances that were overlooked in the original studies because GPs never used a code to record the event. : first published as 0./bmjopen-0-0 on May 0. Downloaded from on April 0 by guest. Protected by copyright. -

8 Page of Each patient was then categorised by a binary variable denoting the style used to record their first five attendances for the feature subsequent attendances were omitted as they were so infrequent. Coded was assigned if any record of a symptom was in coded form; conversely, text-only was designated only when all instances were noted solely in the text. Analysis For each feature, the proportion of attendances attributable to each recording style was determined in event-level analysis. In patient-level analysis, the proportion of attendees attributable to each recording style was determined. Associations between recording style and case/control status were also examined at the patient level for all features in both cancer datasets using the chi-squared test (threshold p-value <0.0). The strength of association between a feature and cancer was assessed using univariable conditional logistic regression. Odds ratios (% CI) are reported before and after addition of previously hidden text records. The chance of cancer in patients presenting with a feature was estimated by the PPV (Bayes Theorem), before and after addition of previously hidden text records.[] All analyses were conducted using Stata (version, StatCorp, College Station, Texas, USA). RESULTS Patient characteristics The participants characteristics are given in Table (see original studies for further detail).[, ] Table Patient characteristics Cancer site Case/control Number (%) in age group: <0 years 0 years Bladder Cases (n=,) (), () Controls (n=,),0 (0), (0) : first published as 0./bmjopen-0-0 on May 0. Downloaded from on April 0 by guest. Protected by copyright. -

9 Page of Pancreas Cases (n=,) (),0 () Controls (n=,), (), () Quantity of data in the hidden text At the event level, considerable numbers of attendances for visible haematuria, jaundice or abdominal pain were recorded solely in the hidden text in both cancer datasets (Figure ). The proportion of records overlooked in the original studies because they were in the hidden text varied with the symptom. In both cancer datasets, approximately half of all attendances for jaundice, and one-third of all attendances for abdominal pain, were recorded solely in hidden text. For visible haematuria, the proportion of hidden text records was greater in the pancreatic (/=%) than in the bladder (,/,0=%) cancer dataset (Figure ). The patient-level data suggested that hidden text was used frequently and consistently, such that the original studies overlooked up to one-third of patients with these features. For example, the proportion of attendees whose entire history of attendance for abdominal pain was recorded solely in hidden text was similar in both cancer datasets (bladder: /,0=%; pancreas: 0/,=%) (Figure ). For alarm features, use of hidden text varied with the feature's strength of association with the cancer subsequently diagnosed in the case. For visible haematuria, records for only /, (0%) of attendees in the bladder cancer dataset were overlooked in the original studies; however, this proportion was doubled (/=%) in the pancreatic cancer dataset. A similar pattern was observed for jaundice recording. Associations between case/control status and recording style of alarm symptoms For visible haematuria in the bladder cancer dataset, the proportion of total attendees whose records were made solely in hidden text was greater in the controls than in the cases (chi-squared test, p<0.00, Figure a, left). However, in the pancreatic cancer dataset, the pattern was reversed in that the proportion of total attendees whose records were made in hidden text was greater in the : first published as 0./bmjopen-0-0 on May 0. Downloaded from on April 0 by guest. Protected by copyright. -

10 Page of cases than in the controls (p<0.0, Figure a, right). This reversal of bias was mainly driven by a large increase in the proportion of cases rendered as hidden text, the proportion of controls overlooked in the hidden text remaining relatively unchanged at approximately 0%. A similar pattern was observed for jaundice recording. The proportion of total attendees whose records were made in hidden text was greater in the controls than in the cases in the pancreatic cancer dataset (p<0.00) (Figure b, left). As above, the bias was reversed in the unconnected cancer. In the bladder cancer dataset, the proportion of total attendees whose records were made in hidden text was smaller in the controls than in the cases (p<0.00, Figure b, right). Associations between case/control status and recording style of non-alarm ( connected, but minor ) symptoms The association between case/control status and recording style is reported for abdominal pain in Figure c. In the pancreatic cancer dataset, hidden text recording affected a greater proportion of controls compared with cases (chi-squared test, p<0.00), although the size of the difference was relatively small (controls: /,=% vs cases: /,=%). In contrast, in the bladder cancer dataset, the records for approximately 0% of controls and cases (p=0.) were made in hidden text (right-hand bars, Figure c), mirroring the pattern seen in pancreatic cancer controls. Effect on odds ratio and positive predictive values Minimal or no overlap of the confidence intervals suggest that PPV values for visible haematuria and jaundice in their associated cancers were reduced by addition of hidden text records (Table ). The OR for bladder cancer in patients presenting with a single episode of visible haematuria tended to be decreased by addition of text-only records, whereas the OR for jaundice was unaffected. Risk estimates for abdominal pain were not altered by addition of hidden text records (Table ). Table Odds ratios (OR) and positive predictive values (PPV) for cancer in symptomatic patients. OR and PPV values were estimated using solely codes (Codes), and using codes plus hidden text records (Total) : first published as 0./bmjopen-0-0 on May 0. Downloaded from on April 0 by guest. Protected by copyright. -

11 Page 0 of Cancer site Symptom OR (% CI) PPV (%)(% CI) Codes Total Codes Total Bladder Visible haematuria 00 ( ) ( ).0 (..). (..) Abdominal pain (..) (..) (0. 0.) (0. 0.) Pancreas Jaundice 0.. Abdominal pain.0 (,) (..) (,) (..) (..) generates detection bias in most CPRD studies, as it is standard practice to restrict analysis to codes. When identifying patients with alarm symptoms, the bias works in favour of those subsequently diagnosed with cancers specific to the alarm symptom, and against those who either do not have, or 0. (..0) 0. (0. 0.) 0. (0.0 0.) Note: The OR was estimated using univariable conditional logistic regression. The PPV was estimated using Bayes' Theorem assuming prior odds of for bladder, and for pancreatic, cancer based on 00 UK national incidence data. DISCUSSION Principal findings This study examined a neglected aspect of utilising CPRD data in symptom-based research; namely, loss of data due to recording style. For large numbers of patients, GPs record attendance for visible haematuria, jaundice or abdominal pain solely in hidden text. Consequently, restricting analysis to codes underestimates symptom prevalence, in patients with and in those without cancer. This confirms and extends our previous findings relating to non-visible haematuria.[] The major new finding of this study is that, for visible haematuria and jaundice, recording style choices lead to differential loss of information in hidden text between cases and controls. This : first published as 0./bmjopen-0-0 on May 0. Downloaded from on April 0 by guest. Protected by copyright. -

12 Page of are diagnosed with an unconnected, cancer. The bias thus inflates PPV values for alarm symptoms in their associated cancers. For abdominal pain, which is more strongly associated with pancreatic than with bladder cancer,[, ] a minor detection bias favours identification of cases with abdominal pain, but only in the pancreatic cancer dataset. Risk estimates are unaffected. The results suggest that GPs are making strong clinical judgements about the probable significance of symptoms preferentially coding clinical features they consider significant to a diagnosis, while tending to use hidden text to record those that they think are not. Strengths and limitations The main strengths of this study are its large size and that the clinical records were made at the time of the consultation, eliminating recall bias. This study like all those based on medical records was limited by reliance on symptom reporting by patients and subsequent recording by GPs. The study of hidden text records brought additional limitations. First, the search criteria will have missed American spellings (e.g. hematuria), and misspellings, though few appeared. Second, some errors are inevitable when classifying text extracts; however, the addition of a manual review of text strings containing words that indicate uncertainty will have minimised these. Finally, the thoroughness of the CPRD s search methods was not documented but, given their emphasis on data quality, we have no concerns about this.[, ] Another limitation of the study is that it did not identify whether the detection bias was caused by the recording behaviour of a small number of GP practices. However, it was important to include data from all contributing practices in order to obtain a measure of bias in CPRD studies overall. This is because the CPRD does not provide quality measures about the quantity of data recorded in hidden text arguably they should be encouraged to do so at the practice level. This is particularly important now that text data are no longer collected and may not be available in future. : first published as 0./bmjopen-0-0 on May 0. Downloaded from on April 0 by guest. Protected by copyright. -

13 Page of Comparison with existing literature Other than our previously published work that identified text records of non-visible haematuria,[] there are no directly comparable studies. Comparisons of numbers of patients overlooked because their records were made in hidden text are complicated by variations in the methods that other researchers used to identify coded records of symptoms. The more comprehensive the code list used to identify instances of symptoms, the greater the prevalence reported, and the smaller the amount of data lost in hidden text. Ford et al. estimated loss of clinical information about rheumatoid arthritis in CPRD hidden text. They identified text-only recording of symptoms of arthritis; for example, codes for synovitis were recorded for of, (.%) patients, but a search of the text suggested that this symptom occurred in, (.%) patients.[] Their study was limited by the omission to identify if the text record was positive (i.e. reporting a current symptom) or 'negative'(i.e. referring to an episode in the past, or confirming the absence of a symptom); therefore, the extent of data lost to analysis is likely to be overestimated. Koeling et al. reported, in conference proceedings, that adding hidden text records increased the percentage of ovarian cancer patients with abdominal pain by a factor of. from % to 0%.[] Our findings for abdominal pain were similar, increasing by factors of. (from /, to 00/,) and. (from,/, to,/,) in bladder and pancreatic cancer cases, respectively. Hayward et al. examined the Consultations in Primary Care Archive database (different to the CPRD, but also with two-tier recording) for breathlessness and wheeze before a diagnosis of ischaemic heart disease, chronic obstructive pulmonary disease or asthma.[] Adding hidden text records increased the percentage of patients with recorded breathlessness or wheeze from 0% to % in cases and from % to % in controls. This suggests a strong association between recording style and patient case/control status (chi-squared test, p<0.00: our estimate), although this was not : first published as 0./bmjopen-0-0 on May 0. Downloaded from on April 0 by guest. Protected by copyright. -

14 Page of directly reported. This result mirrors our findings, and demonstrates that bias arising from recording method is not unique to the CPRD. Implications of the findings The main implication is methodological. Our results will help researchers to estimate detection bias associated with use of hidden text to record symptoms in CPRD studies. Researchers are advised that recording methods may inflate risk estimates for alarm symptoms in their associated diseases, whereas it is not likely to alter risk estimates for non-alarm symptoms. The bias demonstrated in this paper only affects high-risk features. This may not detract from the use of the risk estimates produced by cancer diagnostic studies,[, ] as these features already generally invoke action. Indeed, the corrected PPVs still lie near, or exceed, the % threshold prompting referral for suspected cancer in the latest UK national guidelines.[0] Importantly, 'lowrisk but not no-risk' abdominal pain and non-visible haematuria were subject to little or no bias, as shown in this and our previous study.[] Therefore, their risk estimates are unaltered, and they still meet criteria for inclusion in UK cancer referral guidance, justifying their recent addition.[0] This is particularly reassuring, because low-risk symptoms are common and errors in their risk estimation would have had considerable consequences for the validity of such estimates. Nonetheless, there is a danger that the inflated risk estimates for alarm features will tend to marginalise the significance of low-risk features of cancer. However, it is rare for a low-risk feature to be included in isolation in recommendations for cancer investigation, which typically include the risk of cancer in patients presenting with two symptoms concurrently.[0] Therefore, it is unlikely that detection bias due to loss of records in the hidden text will result in the erroneous omission of a feature from cancer recommendations. Finally, this paper has focussed on cancer, in part because it is one of the main areas where CPRD symptomatic research has concentrated. However, it is reasonable to conclude that our findings may extend to other research areas. There is the strong possibility of underestimation of symptom : first published as 0./bmjopen-0-0 on May 0. Downloaded from on April 0 by guest. Protected by copyright. -

15 Page of frequency possibly coupled with differential recording between comparison groups in all CPRD symptom studies. This does not imply CPRD symptom-based studies are suspect more that they need to be interpreted carefully. ADDITIONAL INFORMATION Ethics approval Independent Scientific Advisory Committee: protocol 0-0. Authors' contributions SJP constructed the dataset, constructed the algorithm to classify text extracts, performed the statistical analysis and drafted the manuscript. SAS and ES conducted the original studies extended in this research. KB and WH participated in the interpretation of text extracts and in the validation of the classification algorithm. WH conceived of the study, and participated in its design and coordination and helped to draft the manuscript. All authors read and approved the final manuscript. No additional data available. REFERENCES Williams T, Tjeerd van S, Puri S, et al. Recent advances in the utility and use of the General Practice Research Database as an example of a UK Primary Care Data resource. Ther Adv Drug Safety 0;():-. Shephard E, Stapley S, Neal RD, et al. Clinical features of bladder cancer in primary care. Br J Gen Pract 0;:e-e0. Stapley S, Peters TJ, Neal RD, et al. The risk of pancreatic cancer in symptomatic patients in primary care: a large case-control study using electronic records. Br J Cancer 0;0():0-. Delgado-Rodriguez M, Llorca J. Bias. J Epidemiol Community Health 00;():-. : first published as 0./bmjopen-0-0 on May 0. Downloaded from on April 0 by guest. Protected by copyright. -

16 Page of Price SJ, Shephard EA, Stapley SA, et al. Non-visible versus visible haematuria and bladder cancer risk: a study of electronic records in primary care. Br J Gen Pract 0;():e-. Shephard E, Neal R, Rose P, et al. Clinical features of kidney cancer in primary care: a casecontrol study using primary care records. Br J Gen Pract 0;(0):e0-. Shephard EA, Neal RD, Rose P, et al. Quantifying the risk of multiple myeloma from symptoms reported in primary care patients: a large case-control study using electronic records. Br J Gen Pract 0;():e0-. Shephard EA, Neal RD, Rose PW, et al. Quantifying the risk of Hodgkin lymphoma in symptomatic primary care patients aged 0 years: a case-control study using electronic records. Br J Gen Pract 0;():e-. Shephard EA, Neal RD, Rose PW, et al. Quantifying the risk of non-hodgkin lymphoma in symptomatic primary care patients aged 0 years: a large case-control study using electronic records. Br J Gen Pract 0;():e-. 0 Stapley S, Peters TJ, Neal RD, et al. The risk of oesophago-gastric cancer in symptomatic patients in primary care: a large case-control study using electronic records. Br J Cancer 0;0():-. Walker S, Hyde C, Hamilton W. Risk of uterine cancer in symptomatic women in primary care: case-control study using electronic records. Br J Gen Pract 0;():e-. Walker S, Hyde C, Hamilton W. Risk of breast cancer in symptomatic women in primary care: a case-control study using electronic records. Br J Gen Pract 0;():e-. Rutjes AW, Reitsma JB, Coomarasamy A, et al. Evaluation of diagnostic tests when there is no gold standard. A review of methods. Health Technol Assess 00;(0):iii, ix-. Holtedahl KA. A method of calculating diagnostic indexes for possible cancer symptoms in general practice. Allgemein Medizin 0;:-. : first published as 0./bmjopen-0-0 on May 0. Downloaded from on April 0 by guest. Protected by copyright. -

17 Page of The General Practice Research Database. GPRD recording guidelines for Vision users. London: Crown Publishing; 00. Herrett E, Gallagher AM, Bhaskaran K, et al. Data resource profile. Clinical Practice Research Datalink (CPRD). Int J Epidemiol 0;():-. Ford E, Nicholson A, Koeling R, et al. Optimising the use of electronic health records to estimate the incidence of rheumatoid arthritis in primary care: what information is hidden in free text? BMC Med Res Methodol 0;():0. Koeling R, Tate AR, Carroll JA. Automatically estimating the incidence of symptoms recorded in GP free text notes. In: First International Workshop on Managing Interoperability and Complexity in Health Systems. Glasgow, Scotland; 0. Hayward RA, Chen Y, Croft P, et al. Presentation of respiratory symptoms prior to diagnosis in general practice: a case-control study examining free text and morbidity codes. 0;():e00. 0 National Institute for Health and Care Excellence. Suspected cancer: recognition and referral [NG]. London: NICE; 0. : first published as 0./bmjopen-0-0 on May 0. Downloaded from on April 0 by guest. Protected by copyright. -

18 Page of Figure Event-level analysis: number of attendances for visible haematuria, jaundice and abdominal pain in bladder and pancreatic cancer dataset grouped by recording method (coded, or solely in the text) Figure Patient-level analysis: number of attendees for visible haematuria, jaundice and abdominal pain in bladder and pancreatic cancer datasets grouped by recording method (coded, or solely in the text) Note: At the patient level, patients were categorised as having a text-only recording style if all of their attendances were documented solely in the text. Figure a-c Patient-level analysis. a Number of attendees for visible haematuria in the bladder (left) and pancreatic (right) cancer datasets grouped by recording method (coded, or solely in the text) and patient status (case or control). b Number of attendees for jaundice in the pancreatic (left) and bladder (right) cancer datasets grouped by recording method (coded, or solely in the text) and patient status (case or control). c number of attendees for abdominal pain in the pancreatic (left) and bladder (right) cancer datasets grouped by recording method (coded, or solely in the text) and patient status (case or control) Note: *** = p<0.00; * 0.0>p>0.0. : first published as 0./bmjopen-0-0 on May 0. Downloaded from on April 0 by guest. Protected by copyright. -

19 Page of xmm (00 x 00 DPI) - : first published as 0./bmjopen-0-0 on May 0. Downloaded from on April 0 by guest. Protected by copyright.

20 Page of xmm (00 x 00 DPI) - : first published as 0./bmjopen-0-0 on May 0. Downloaded from on April 0 by guest. Protected by copyright.

21 Page 0 of xmm (00 x 00 DPI) - : first published as 0./bmjopen-0-0 on May 0. Downloaded from on April 0 by guest. Protected by copyright.

22 Page of xmm (00 x 00 DPI) - : first published as 0./bmjopen-0-0 on May 0. Downloaded from on April 0 by guest. Protected by copyright.

23 Page of xmm (00 x 00 DPI) - : first published as 0./bmjopen-0-0 on May 0. Downloaded from on April 0 by guest. Protected by copyright.

24 Page of StroBe statement checklist of items that should be included in reports of observational studies Item No Recommendation Title and abstract Item Recommendation Followed? Page no. number (a) Indicate the study s design with a commonly used term in the title or Within the title Within the abstract - the abstract (b) Provide in the abstract an informative and balanced summary of what was done and what was found Yes. - Introduction Background/rationale Explain the scientific background and rationale for the investigation being reported Throughout the Introduction, Objectives State specific objectives, See in particular the final including any prespecified hypotheses sentence of the Introduction Methods Study design Present key elements of Paragraph study design early in the paper Setting Describe the setting, locations, and relevant dates, including periods of recruitment, exposure, follow-up, and data collection See section Setting, patients and period of study Participants Case-control study Give the See section Setting, eligibility criteria, and the sources and methods of patients and period of study case ascertainment and control selection. Give the rationale for the choice of cases and controls Case-control study For matched studies, give matching criteria and the number of controls per case See section Setting, patients and period of study See also Results, Table Readers are also referred to the original studies extended in my study, where full details are given Variables Clearly define all outcomes, exposures, predictors, potential confounders, and effect modifiers. Give See section 'Variables' in Methods - : first published as 0./bmjopen-0-0 on May 0. Downloaded from on April 0 by guest. Protected by copyright.

25 Page of Data sources/ measurement diagnostic criteria, if applicable * For each variable of interest, give sources of data and details of methods of assessment (measurement). Describe comparability of assessment methods if there is more than one group See sections Variables' and 'Analysis' in Methods. In the first sentence of the Methods, readers are also referred to previous paper, which contains all this information on page e in the section Selection of investigations and symptom variables. See all sections of the Methods. Bias Describe any efforts to address potential sources of bias Study size 0 Explain how the study size was arrived at Quantitative variables Explain how quantitative variables were handled in the analyses. If applicable, describe which groupings were chosen and why Statistical methods (a) Describe all statistical methods, including those used to control for confounding (b) Describe any methods used to examine subgroups and interactions (c) Explain how missing data were addressed Case-control study If applicable, explain how matching of cases and controls was addressed In the first sentence of the Methods, readers are referred to the previous papers, which describe (on pages e00 Bladder) that a power calculation was carried out instead of a study size calculation because this is a casecontrol study in which the number of cases is fixed. See section 'Variables' and 'Analysis',,,, - See section 'Analysis' - n/a See section Variables' - See section 'Setting, patients and period of study'. Also, in the first sentence of the Methods, readers are referred to the original papers extended by this study, where all such information can be found (bladder: page e in the - : first published as 0./bmjopen-0-0 on May 0. Downloaded from on April 0 by guest. Protected by copyright.

26 Page of Results Participants Descriptive data (e) Describe any sensitivity analyses * (a) Report numbers of individuals at each stage of study eg numbers potentially eligible, examined for eligibility, confirmed eligible, included in the study, completing follow-up, and analysed (b) Give reasons for nonparticipation at each stage section Cases and controls ; pancreas: p, 'Identification of cases and controls'. None completed In the first sentence of the Results, readers are referred to the original studies, which contain all this information (bladder: page e00 in the first paragraph of Results; pancreas: page, Figure ). In the first sentence of the Results, readers are referred to the original studies, which contain all this information (bladder: page e00 in the first paragraph of Results; pancreas: page, Figure ). Decided against this, as these data are reported in the original studies Table. Also in the first sentence of the Results, readers are referred to the original papers, which contain all this information (bladder: pages e00, e0 and Table ; pancreas: p, Table ). Figures,, and (missing data are assigned 'symptom absent') (c) Consider use of a flow diagram * (a) Give characteristics of study participants (eg demographic, clinical, social) and information on exposures and potential confounders (b) Indicate number of participants with missing data for each variable of interest Outcome data * Case-control study Report numbers in each exposure category, or summary measures of exposure Main results (a) Give unadjusted estimates and, if applicable, confounder-adjusted estimates and their precision (eg, % confidence interval). Make clear which confounders were adjusted for and why they were Figures,, and Table - n/a : first published as 0./bmjopen-0-0 on May 0. Downloaded from on April 0 by guest. Protected by copyright.

27 Page of included (b) Report category boundaries when continuous variables were categorized (c) If relevant, consider translating estimates of relative risk into absolute risk for a meaningful time period Other analyses Report other analyses done eg analyses of subgroups and interactions, and sensitivity analyses n/a n/a Patient-level data Figures, & Discussion Key results Summarise key results with Section 'Summary' of 0 reference to study Discussion objectives Limitations Discuss limitations of the Section Strengths and 0, study, taking into account sources of potential bias or imprecision. Discuss both direction and magnitude of any potential bias limitations Interpretation 0 Give a cautious overall See Comparison with - interpretation of results considering objectives, limitations, multiplicity of analyses, results from similar studies, and other relevant evidence existing literature Generalisability Discuss the generalisability See Implications of the - (external validity) of the study results findings' Other information Funding Give the source of funding and the role of the funders for the present study and, if applicable, for the original study on which the present article is based No specific funding n/a *Give information separately for cases and controls in case-control studies and, if applicable, for exposed and unexposed groups in cohort and cross sectional studies. The STROBE checklist is best used in conjunction with the explanation and elaboration article. -0 This article and separate versions of the checklist for cohort, case-control, and cross sectional studies are available at n/a n/a : first published as 0./bmjopen-0-0 on May 0. Downloaded from on April 0 by guest. Protected by copyright.

28 Is omission of free text records a possible source of data loss and bias in Clinical Practice Research Datalink studies? A case control study Journal: Manuscript ID bmjopen-0-0.r Article Type: Research Date Submitted by the Author: 0-Apr-0 Complete List of Authors: Price, Sarah; University of Exeter Medical School Stapley, Sal; University of Exeter Medical School Shephard, Elizabeth; University of Exeter Medical School Barraclough, Kevin; Hoyland House Hamilton, Willie; University of Exeter Medical School <b>primary Subject Heading</b>: Epidemiology Secondary Subject Heading: General practice / Family practice Keywords: PRIMARY CARE, STATISTICS & RESEARCH METHODS, ONCOLOGY : first published as 0./bmjopen-0-0 on May 0. Downloaded from on April 0 by guest. Protected by copyright. -

29 Page of Is omission of free text records a possible source of data loss and bias in Clinical Practice Research Datalink studies? A case control study Sarah J. Price*: Medical School, University of Exeter, College House, St Luke s Campus, Exeter, UK, EX LU, tel: +--, S.J.Price@exeter.ac.uk Sal A. Stapley: Medical School, University of Exeter, College House, St Luke s Campus, Exeter, UK, EX LU, S.A.Stapley@exeter.ac.uk Elizabeth Shephard: Medical School, University of Exeter, College House, St Luke s Campus, Exeter, UK, EX LU, E.A.Shephard@exeter.ac.uk Kevin Barraclough:Hoyland House, Gyde Road, Painswick, Gloucestershire, UK, GL RD, UK, k.barraclough@btinternet.com William T. Hamilton: Medical School, University of Exeter, College House, St Luke s Campus, Exeter, UK, EX LU, W.Hamilton@exeter.ac.uk *Correspondence to: Sarah J. Price, Medical School, University of Exeter, College House, St Luke s Campus, Exeter, UK, EX LU, tel: +--, S.J.Price@exeter.ac.uk Key words: Cancer, Disease modelling, General practice, Methodology, Research design in epidemiology Word count: : first published as 0./bmjopen-0-0 on May 0. Downloaded from on April 0 by guest. Protected by copyright. -

30 Page of ABSTRACT Objectives: To estimate data loss and bias in studies of Clinical Practice Research Datalink (CPRD) data that restrict analyses to Read codes, omitting anything recorded as text. Design: Matched case control study. Setting: Patients contributing data to the CPRD. Participants:, bladder, and, pancreatic, cancer cases diagnosed between 0/0/000 and //00, matched on age, sex and GP practice to up to controls (bladder: n=,; pancreas: n=,). The analysis period was the year before cancer diagnosis. Primary and secondary outcome measures: Frequency of haematuria, jaundice and abdominal pain, grouped by recording style: Read code or text-only (i.e. hidden text). The association between recording style and case/control status (chi-squared test). For each feature, the odds ratio (conditional logistic regression) and positive predictive value (PPV; Bayes' Theorem) for cancer, before and after addition of hidden text records. Results: Of the 0, total records of the features,, (%) were recorded in hidden text. Hidden text recording was more strongly associated with controls than with cases for haematuria (0/=% vs /,=%) in bladder cancer (chi-squared test, p<0.00), and for jaundice (/=% vs /,=0%, p<0.000) and abdominal pain (/,=% vs /,=%, p<0.00) in pancreatic cancer. Adding hidden text records corrected PPVs of haematuria for bladder cancer from.0% (%CI:..%) to.% (..%) and of jaundice for pancreatic cancer from.% (..%) to.% (..%). Adding hidden text records did not alter the PPV of abdominal pain for bladder (codes: 0.%, 0. 0.% vs codes plus hidden text: 0.%, 0. 0.%) or pancreatic (0.%, 0. 0.% vs 0.%, %) cancer. : first published as 0./bmjopen-0-0 on May 0. Downloaded from on April 0 by guest. Protected by copyright. -

31 Page of Conclusions: Omission of text records from CPRD studies introduces bias that inflates outcome measures for recognised alarm symptoms. This potentially reinforces clinicians views of the known importance of these symptoms, marginalising the significance of low-risk but not no-risk symptoms. ARTICLE SUMMARY Strengths and limitations of this study Set in the CPRD, in which electronic medical records are recorded prospectively, eliminating recall bias. Large study size. Comprehensive identification of coded records of cancer symptoms. Search criteria for identifying text records of cancer symptoms will have missed US spellings and spelling errors. Study did not identify whether text records arose through the recording practices of a small number of GP practices. FUNDING STATEMENT Sarah J Price is funded by a University of Exeter PhD studentship. COMPETING INTERESTS The author(s) declare that they have no competing interests. INTRODUCTION The Clinical Practice Research Datalink (CPRD) is a UK-based research service that maintains a database of anonymised copies of medical records collected by general practitioners (GPs) as part of everyday clinical care. The database is the largest of its kind and a representative sample of the UK population. Though created for pharmacological research, the database is now used extensively in epidemiological studies.[, ]Quality standards for the CPRD devised for pharmaco-epidemiology research have not been amended to accommodate areas such as symptom-based research. One potential, but neglected, concern for those using CPRD data arises from the way clinical events are recorded in the GP surgery. Most GP practices that contribute data to the CPRD use ViSion : first published as 0./bmjopen-0-0 on May 0. Downloaded from on April 0 by guest. Protected by copyright. -

32 Page of software (ViSion INPS, London, UK), in which GPs must choose a Read code to begin a record. Read codes have been used in the National Health Service to record patient findings since, and there are now over,000 of them, organised in a hierarchical classification system. Once a Read code has been chosen, a comments box opens in which GPs can type freely and are not limited to elaborating on the code. While codes are fully and routinely available to researchers, text records are not. Furthermore, a moratorium on the collection of text data by the CPRD was introduced in 0. Text recorded before 0 can be accessed by researchers, but this is rarely done because the methods are complex, expensive and limited. Therefore, to be detected by researchers, clinical information in the medical record must be recorded using a Read code. Consequently, the omission of text records may introduce a 'detection' bias, because researchers are oblivious to clinical events that are only ever recorded as an inaccessible text comment (henceforth called 'hidden text').[] In a previous CPRD case control study, we identified that, of patients with non-visible haematuria, records of this fact were made solely in hidden text for (%). The proportional loss of records in the hidden text was similar in cases and controls. Therefore, the effect of the detection bias was limited to underestimation of the frequency of non-visible haematuria.[] This paper further investigates the potential consequences for research of having two recording methods codes and hidden text. Specifically, we sought evidence for the disproportionate use of hidden text to record symptoms/signs presented by cases compared with controls, as this will result in biased estimates of outcome measures such as odds ratios (ORs) and positive predictive values (PPVs). This is important, because the outputs of such research are used as evidence to underpin national guidelines; for example, estimates of cancer risk in symptomatic patients presenting to primary care.[,] METHOD : first published as 0./bmjopen-0-0 on May 0. Downloaded from on April 0 by guest. Protected by copyright. -