D5.4 Specification for functional ecrf and DSS

Size: px
Start display at page:

Download "D5.4 Specification for functional ecrf and DSS"

Transcription

1 Translational Research and Patient Safety in Europe Work Package Number: Work Package Title: WP5 Interfaces: User and Software Services Nature of Deliverable: Other Dissemination Level: Public Version: v1.0 Delivery Date From Annex 1: M51 (May 2014) Principal Authors: Olga Kostopoulou (KCL), Talya Porat (KCL), Lei Zhao (UW), Sarah N. Lim Choi Keung (UW), Theodoros N. Arvanitis (UW) Contributing Authors: Samhar Mahmoud (KCL), Piotr Bródka (WRuT), Andrzej Misiaszek (WRuT), Wlodzimierz Tuligłowicz (WRUT), J.-F. Ethier (INSERM), Anna Andreasson (KI) Partner Institutions: This project has received funding from the European Union s Seventh Framework Programme for research, technological development and demonstration under grant agreement no [TRANSFoRm]. 1

2 Table of Contents List of Figures... 3 List of Tables... 4 Abbreviations... 5 Executive Summary Specification for functional DSS Introduction User Interface Requirements and Recommendations DDSS Design Usability Evaluation of the dummy prototype Summary References Specification of a functional ecrf to be integrated with an EHR Overview Semantic-aware ecrf Modelling GORD Evaluation Study Specification Conclusion References Appendix 1: User Requirements Elicitation (manuscript) Appendix 2: DDSS User Guide Appendix 3: GORD-SDM.xml Appendix 4: Study Data Collection Forms Appendix 4A CROMs Appendix 4B PROMs

3 List of Figures Figure 1: GORD study timeline [2, Figure 2] Figure 2. CDISC suite of standards [9] Figure 3. Structure of the XML file Figure 4. Example of Treatment Epoch element for the GORD study Figure 5. Example of Continuous Arm element for the GORD study Figure 6. Example of Treatment cell for the GORD study Figure 7. Example of the Follow-up segment for the GORD study Figure 8. Examples of the criteria check and study statistics collection activities in the GORD study Figure 9. Structural elements of the GORD study Figure 10. Activities with workflow-specific roles in the GORD study Figure 11. Examples of Transition and Switch elements for the eligibility criteria check activity Figure 12. Example of a Trigger element in the GORD study Figure 13. Timing constraint for the first visit PROM collection in the continuous arm Figure 14. ODM file attributes for the GORD study Figure 15. StudyEventDef element for unscheduled visits in the on-demand arm Figure 16. FormRef element for an event visit CROM Figure 17. ItemGroupDef element for the item group for weight Figure 18. ItemDef element for the weight item Figure 19. Link between ODM and the Reference Model via archetypes Figure 20. Part of the data extraction query for weight

4 Figure 21. Archetype for weight - definition Figure 22. Archetype for weight - ontology definitions Figure 23. Archetype for weight - ontology term bindings Figure 24. Archetype-based query model main elements Figure 25. Query model - expressions Figure 26. GORD evaluation study workflow Figure 27. Randomisation blocking definition Figure 28. Study parameter planned number of recruited subjects List of Tables Table 1. EHR data elements for CROM form pre-population TABLE 2: Excerpt from the SAR analysis of a think-aloud protocol of GP 9 (cancer diagnosis study) TABLE 3: Excerpt from the SAR analysis of a CDM protocol of GP 8 (intuition study) TABLE 4: Cognitive requirements of the diagnostic process

5 Abbreviations CDIM Clinical Data Integration Model CRIM Clinical Research Information Model CROM Clinician Reported Outcome Measures DNC Data Node Connector DDSS Diagnostic Decision Support System DSS Decision Support System EC Eligibility Criteria ecrf electronic Case Report Form EHR Electronic Health Record GORD Gastro-oesophageal reflux disease GP General Practitioner / General Physician ODM Operational Data Model PRM Protocol Representation Model PROM Patient Reported Outcome Measures RfE Reason for Encounter SDB Study Database SDM Study Design Model SDTM Study Data Tabulation Model SF-12 Short Form 12 TAM Technical Acceptance Model TSS TRANSFoRm Study System VarQs Form variables expressed using the TRANSFoRm query model 5

6 Executive Summary This document delivers the functional specifications for ecrf and DSS TRANSFoRm software configurations. It is composed of two sections. Section 1 focuses on the DSS and provides: (1) a summary of the diagnostic decision support system (DDSS) design requirements and recommendations (2) a short description on the proposed user interface following the requirements and recommendations and (3) main findings of the preliminary prototype usability evaluation. We are currently updating the design of the prototype according to the usability findings and we will continue to evaluate and update the DDSS using participatory design methods. A manuscript on the user requirements elicitation methods and design recommendations is included in Appendix 1: User Requirements Elicitation (manuscript) and is currently being prepared for submission to a peer-reviewed journal). a detailed DDSS user guide is included in Appendix 2: DDSS User Guide. Section 2 of this deliverable describes the outcomes of WT 5.2a Specification for a functional ecrf to be integrated with an EHR. It reports the development of the specification and demonstration of a functional electronic Case Report Form, designed to enable the collection of semantically-controlled data from within an EHR. The ecrf model described specifies the way ecrf forms are constructed, and how the ecrf data is made semantically interoperable with clinical data from within an EHR. The specification of the ecrf model is described using the example of the TRANSFoRm GORD validation study. Following the model-based approach in TRANSFoRm, the ecrf model works within the scope of the Clinical Research Information Model (CRIM), D6.2. The model describes the lifecycle of the study data collection process, including activities such as informed consent and randomisation, as defined in the study protocol. It also supports data collection using various methods (mobile, web interface, software within the EHR system), in various languages, and is in a format that can be deployed within an EHR system (e.g. XML format). The specification also shows how the CDISC Standard Operational Data Model (ODM) is extended to enable ecrf data to be semantically interoperable with EHR 6

7 data. This is mainly to enable EHR data to be extracted and pre-populate ecrfs where applicable. The deliverable demonstrates how archetypes representing clinical data elements, linked to the Clinical Data Integration Model (CDIM) ontology (D6.3), are bound to ecrf form elements to maintain the semantic meaning throughout. The extended ODM document that is exchanged between the Study System and an EHR system will include queries describing how ecrf form elements can be extracted from the EHR system. 7

8 1. Specification for functional DSS 1.1 Introduction A major part of TRANSFoRm is the design and development of a functional prototype for a diagnostic decision support system (DDSS) that will be integrated with the Electronic Health Record (EHR), for aiding general physicians in the diagnosis of three major reasons for encounter: chest pain, abdominal pain and dyspnoea. In the future, the DDSS can be expanded to include other reasons for encounter, though this is beyond the scope of TRANSFoRm. This document summarises the user interface requirements, design and evaluation of the DDSS prototype. For further information, please refer to the appendices. 1.2 User Interface Requirements and Recommendations To elicit user requirements for the design of the DDSS, we conducted in situ observations and interviews of 8 family physicians consulting with patients. We also used data collected in other recent studies of Dr Olga Kostopoulou. Specifically, we analysed 34 think-aloud protocols of 17 physicians diagnosing computerised patients, and 24 interview protocols of 18 family physicians describing past cases of intuitive diagnosis. Transcripts were coded using the Situation Assessment Record method (SAR; Hoffman et al., 1998). The result of the analysis was a Decision Requirements Table (See Appendix 1: User Requirements Elicitation (manuscript)) that guided the user interface design recommendations for the DDSS. Below are the main design recommendations: Highlight important patient information existing on the health record, e.g. risk factors. Display possible diagnoses at the start of the clinical encounter, based on existing information in the health record and the current reason for encounter (RfE). Enable users to select a diagnosis from the list and check what features (symptoms and signs) are relevant (i.e. can impact on the likelihood of that diagnosis). Enable users to indicate both the presence and absence of symptoms and signs. 8

9 Facilitate data insertion and coding during the consultation. Propose specific and relevant examinations and investigations that could be preformed in order to differentiate between the diagnoses on the list. Alert physicians after they give a diagnosis but have failed to check any symptoms and signs of serious or urgent possible diagnoses. See Appendix 1: User Requirements Elicitation (manuscript) for the manuscript in preparation that describes the user requirements elicitation methods and user interface design recommendations. See DELIVERABLE 4.1 for the technical and functional aspects of the DDSS. 1.3 DDSS Design Based on the above requirements and recommendations we tried several, preliminary user interface designs of the DDSS. We presented and discussed these at TRANSFoRm face-to-face meetings and with GPs from the department of primary care of King s College London. On the basis of these discussions, a design was adopted and a dummy prototype was developed and evaluated with 10 experienced GPs (see 1.4 Usability Evaluation of the dummy prototype). 9

10 Appendix 2: DDSS User Guide presents the DDSS User Guide that explains the DDSS functions and how GPs can use it. For the final evaluation study of the fullyintegrated and functioning DDSS prototype (WT3), we have an agreement with Vision EHR ( to integrate the DDSS into their system, though the DDSS could be integrated with any EHR system. 1.4 Usability Evaluation of the dummy prototype For the usability evaluation, a dummy interactive prototype of the DDSS was developed. The prototype is dummy in the sense that it is not yet integrated with Vision EHR or the evidence database. Hence, the selections (e.g., RfE, signs, symptoms, examinations, and diagnosis) and the suggested diagnoses list are predefined and static (i.e. do not change according to user input). See the DDSS Interactive Demo demonstrating the prototype functionality Method We evaluated the dummy prototype with 10 experienced GPs (6 male, mean 13.4 years in general practice, range 4 to 23 years). Our aim was to evaluate the three usability goals: effectiveness (can users use the tool?), efficiency (how well can they use it?) and satisfaction with the tool (ISO ). Each evaluation lasted up to 1.5 hours. Using the DDSS prototype, participants were asked to diagnose two scenarios of young women presenting with abdominal pain. The researcher simulated the two patients but did not interrupt participants in any other way. She observed and noted the participants workflow, selections on the screen, the information that they entered in the record, errors in the recording of information and other key incidents, and comments that they made. At the end of each scenario and at the end of the evaluation, the researcher interviewed the participants. The interview addressed participants overall impressions of the diagnostic system and specifically asked what they liked or disliked about it, what features they found useful, and to name situations where they thought that the system would be most effective. The interview also asked what additional functionality or improvements they would like and whether they would like to use this tool in the future. Finally, the researcher sought opinions on a specific design idea: a premature closure alert, which would be triggered after a diagnosis is entered and only if the GP has not checked any of the symptoms and signs of 10

11 serious or urgent possible conditions Main findings Effectiveness All participants managed to enter the patient s main RfE at the beginning of the consultation and the main signs, symptoms and examination results during the consultation. All participants viewed the list of suggested diagnoses and interacted with it (i.e., clicked on a diagnosis) Efficiency The diagnostic system was very easy to learn and use appropriately. All participants managed to use it after 3 minutes of explanation and approximately 2 minutes of system trial. No errors were performed using the tool, except for one participant who made it quit by mistake (clicked the Done button before entering all the patient information) Satisfaction Perceived Effectiveness Participants were enthusiastic about the system. They thought it could be very effective and help them in the diagnostic process. Particularly, they thought it could expand the number of hypotheses they consider and prevent them for missing important information. I didn t think about Pelvic Inflammatory Disease, however I saw it on the list, so I wanted to rule it out (GP 2). They liked the idea that the system suggests possible diagnoses based on information from the patient s health record and enables physicians to view main signs and symptoms relating to a specific diagnosis. Since I thought she has bacterial infection, I clicked on it and ticked all the signs, very helpful (physician 7). The premature closure alert was presented and demonstrated to participants at the end of the evaluation, asking their opinion about its possible effectiveness. All participants thought that it could be very useful, both for helping them not to miss serious diagnoses and also for legal reasons (making sure they ask appropriate questions for serious or urgent diagnoses). Excellent idea, just by itself its great (physician 2); This tool makes me think about serious diagnoses that I might miss (physician 6); For legal reasons its great, because you can show that you thought 11

12 about these serious things, I would really like it (physician 4). Participants raised two main concerns relating to the DDSS effectiveness: 1. Most participants said they would like to use the system in relatively complex, rather than straightforward cases. I think in the Appendicitis case the tool is not very useful, it will be in more complex cases, like the Crohn s scenario (physician 2). In addition, they think that the three reasons for encounter the system currently supports (abdominal pain, chest pain, dyspnoea) are effective, but they would like to elaborate the use of the tool to other conditions as well, such as: headaches, leg pains, cough, anaemia, and weight loss. 2. Two GPs raised a concern that the patients may not like the idea that the GP uses such a tool: Patients won t like GPs to use it (physician 4). However, most participants were not concerned about this, even one of the GPs that raised the concern gave a solution: I will answer in the same way as I answer when they ask me why I use GP notebook, I say: I am just double checking or looking at the latest guidelines, and now I can say: I just want to make sure I am not missing any serious diagnoses, and then its OK, patients won t mind (physician 2). Perceived Efficiency Participants felt that the system was very easy to use and they had no difficulties learning to use it and entering the information while interacting with the simulated patient (i.e., researcher). Specifically, participants liked the opportunity to record absence of symptoms and signs as well as presence and to have a summary of all the information that they gathered, including examination results: The system focuses me on the task of diagnosing, I know where I am and what I have to do, what I already asked and what I still need to check, sometimes in my work there are so many distractions that I don t remember why the patient even came (physician 3). Participants understand the importance of coding information and thought that this system could motivate them to code more than they currently do. The main concern is that coding information into the DDSS may be time consuming, and therefore ways to improve data coding and insertion should be considered. 12

13 1.4.3 Proposed changes In view of the above findings, two main changes are proposed to the DDSS prototype: 1. Add a confirmation dialog to prevent users from closing the DDSS by mistake (e.g., when clicking the Done button or the X icon). 2. Facilitate data insertion and coding to decrease interaction time in the following ways: Support use of the keyboard (in addition to the mouse) throughout the system, to reduce interaction time, Facilitate entry of examination results by adding designated fields close to the selected examination (e.g., select temperature and enter the value). If the user selected a specific examination (e.g., examination of abdomen or chest examination), the system will display a group of test results that are relevant for the examination. For example, if the user selects: examination of abdomen, the system will provide relevant examination results (e.g., normal, tenderness, guarding, rebound, etc.) and the user will just tick Yes or No. Most participants in the usability evaluation coded examination of abdomen, but wrote the examination results in free text. This solution will facilitate coding the examination results and will provide the DDSS with important information that can be used to differentiate between the possible diagnoses. In future versions, as proposed by us and by one of the participants, patients may enter the RfE and answer questions about symptoms, while waiting for the physician. The DDSS could then take this information into account when suggesting possible diagnoses. This solution can reduce the amount of time the GP interacts with the system. 1.5 Summary Section 1 summarises the design process of the DDSS from the elicitation of user requirements to the design of a dummy prototype and its preliminary usability evaluation. Initial findings are promising and suggest that the system has the potential of being both effective and efficient. Iterative design is important and we will continue to update and evaluate the DDSS, and plan more usability evaluations with GPs once the prototype will be integrated to 13

14 Vision EHR and to the evidence database. 1.6 References 1. Hoffman, R. R., Crandall, B. W., & Shadbolt, N. R. (1998). Use of the critical decision method to elicit expert knowledge: A case study in cognitive task analysis methodology. Human Factors, 40(2), ISO Ergonomic requirements for office work with visual display terminals (VDTs) Part 11: guidance on usability. International Organization for Standardization (1998). (additional references in the Appendix 1 and 2) 14

15 2. Specification of a functional ecrf to be integrated with an EHR 2.1 Overview A core output of the TRANSFoRm project is the specification and demonstration of a functional electronic Case Report Form (ecrf) [1, p. 21, WT5.2a]. The ecrf is designed to enable the collection of semantically-controlled and standardised data from within an EHR system. In the context of TRANSFoRm, the ecrf specification needs to satisfy the following requirements: The ecrf specification will support the collection of semantically controlled data. Primary care patient data in European EHR systems are generally coded with specific clinical coding schemes such as ICPC [2], ATC [3], etc. The ecrf specification should enable the collection of EHR data coded in different coding schemes from various EHR systems in different countries. The development of the ecrf specification will follow a model-based approach in compliance with the project philosophy of TRANSFoRm. An ecrf model will be developed within the framework established by Clinical Research Information Model (CRIM) [4]. The ecrf model will enable semantic interoperability between the study system and the EHR system, and support automated pre-population of EHR data into study case report forms. The ecrf model will support the collection of data through a variety of technologies: web interface, mobile application, as well as data collection software embedded in an EHR system. The ecrf specification will support international languages in addition to English. So when deployed in different countries, the case report forms will present native languages to users. The ecrf model will cover the whole lifecycle of study data collection, and include support for various study activities such as eligible patient identification and recruitment, informed consent, study id allocation, randomisation, adverse event monitoring, etc. The specification should have appropriate representations (such as XML) for 15

16 deployment within an EHR system. The ecrf model will be closely aligned to existing standards produced by CDISC [5] and will be taken to CDISC for incorporation into standards. This is essential to enable interoperability between TRANSFoRm and Clinical Trial Data management and electronic remote data capture systems. The ecrf model will be validated by TRANSFoRm GORD use case (D1.1). An instance specification is developed for the GORD evaluation study. This study specification will be tested in the TRANSFoRm federated infrastructure which manages the deployment of the functional ecrfs in EHR systems (D7.1). The main aim of this work is to make the ecrf forms and the data collected on them semantically interoperable with data from EHR systems. EHR data can be used to pre-populate ecrfs. In the following sections, we will recapitulate the GORD use case briefly and then describe in detail our ecrf modelling methodology in section 2.2 Semantic-aware ecrf Modelling. The GORD use case will be used throughout that section as one illustrative example to elaborate the technical details. Finally, section 2.3 GORD Evaluation Study Specification details how the model of GORD study is constructed by applying the modelling methodology presented in section 2.2 Semantic-aware ecrf Modelling GORD Use Case The gastro-oesophageal reflux disease (GORD) use case was introduced in deliverable D1.1[6]. This research use case is being used as a representative randomised control trial. GORD is a spectrum of disorders mainly caused by the retrograde flow of gastric acid from the stomach into the oesophagus. The main symptoms are heartburn and acid regurgitation. GORD is treated using proton pump inhibitors (PPI), H2-blockers or antacids. As many patients still have symptoms in spite of PPI treatment, continuous versus on-demand PPI use regarding symptom severity and quality of life is being compared in this study. GORD Evaluation Study The aim of the evaluation study is to: (1) determine the effectiveness in terms of symptom control and quality of life (QoL) and event-initiated assessment of QoL and symptoms in patients with GORD, randomised to either continuous or on-demand use of PPI; and (2) to investigate whether using an electronic system to recruit study 16

17 participants and to collect data increases the recruitment rate in primary care clinical studies [7], [8]. In this clustered randomised controlled trial, practices will be randomised to either using standard trial methods (paper version) or using TRANSFoRm s electronic tools. In the TRANSFoRm supported arm, the GORD study will have the following timeline (Figure 1) and data collection workflow: Figure 1: GORD study timeline [2, Figure 2] Visit 1: The patient attends the primary care practice, either already taking PPI, or as a new reflux case with a diagnosis of GORD or a symptom of GORD. At this point, the patient will be flagged as potentially eligible for the study. The general practitioner (GP) checks the eligibility criteria and gives study information if the patient is eligible. If the patient consents to take part in the study, the patient is randomised to either continuous or on-demand PPI. o CROMs are completed by the GP in the ecrf. o GP is informed of the randomisation outcome. o PROMs are completed by the patient via their smartphone (via Android or ios mobile application) or computer (via web browser and web application). Visit 2: After 8 weeks, the patient returns for a follow-up visit. CROMs and PROMs are completed by the GP and patient respectively. Event: If the patient visits the practice between Visit 1 and Visit 2, CROMs for event visits are completed by the GP. 17

18 At the end of the study, participants and GPs will be asked to answer some question on the acceptance of the TRANSFoRm system (TAM survey). Study Data Collection Forms All the forms that will be used for data collection (CROMs and PROMs) are available for reference in Appendix 4: Study Data Collection Forms Appendix 4A CROMs Paper Version electronic Case Report Forms (CROMs) Appendix 4B PROMs Patient Reported Outcome Measures (PROMs) 2.2 Semantic-aware ecrf Modelling Modelling Philosophy The development of ecrf specification follows a model-based approach in compliance with the project philosophy of TRANSFoRm. The modelling process is based on those standard information models widely accepted in the health informatics domain. Following is a short summary of the modelling process. Study timeline is modelled by CDSIC SDM. Case report forms are modelled by CDISC ODM. Clinical information available in EHR systems can be used to answer questions in the case report forms. Automated data collection from EHR systems requires computable definitions of those clinical data elements. The openehr archetype approach is used as the main mechanism to model EHR data. Based on the archetype approach, a query model is developed to enable formulation of eligibility criteria and data extraction requests, which can be interpreted and executed by EHR systems. CDISC ODM is extended to embed data extraction query requests, so the ecrfs can be pre-populated using the EHR data when they are deployed into an EHR system. EHR systems across Europe use heterogeneous systems to record and manage clinical data in primary care. Clinical concepts can be semantically interoperable via common ontologies, and mappable medical vocabularies, as has been done in the TRANSFoRm project with the CDIM ontology and the TRANSFoRm vocabulary 18

19 service. In TRANSFoRm, the clinical data from EHR systems and data warehouses need to be additionally semantically interoperable with other systems, such as the study system, in particular the ecrf forms and the data that are collected within them, that are common to both the clinical and research domains. ecrfs are commonly structured according to standards, such as CDISC. There is a need to make data elements that are common to both domains semantically interoperable. This is to ensure that data can be extracted from EHR systems to populate ecrfs, and also to report ecrf data back into the EHR systems to assist medical practitioners. In TRANSFoRm, this semantic interoperability between the clinical and research domains is enabled by semantically extending the CDISC ODM standard to maintain the meaning of clinical data elements between systems crossing these two domains, as will be described in the rest of this document. The CDISC foundational standards form the basis of a suite of standards supporting the clinical research process from protocol through data collection, data management, data analysis and reporting [9]. They represent interest areas that are common across research studies, such as demographics, medication history, medical history, adverse events. The foundational standards are shown at the top of Figure 2. 19

20 Figure 2. CDISC suite of standards [9] The file format that will be used to exchange ecrf data between the TRANSFoRm Study System and the EHR systems is SDM as an XML document. The document can be schematically depicted as in Figure 3 and the following sections will describe each section in more detail. 20

21 Metadata definitions Study design Queries and archetypes Figure 3. Structure of the XML file CDISC SDM The clinical research study protocol is the plan that describes the study s objectives, methodology, statistical considerations, and the organisation of the study [10]. This plan includes the design of the study, which includes the arm descriptions, the schedule of activities, the eligibility criteria and summary information [11], [12]. Several CDISC standards represent aspects of the study design, but do not specify the study design completely. For instance, the Operational Data Model (ODM) represents the metadata for the data collected in the study, but does not describe the planned timing of the study events (ODM is described further in Section CDISC ODM). The Study Data Tabulation Model (SDTM) includes trial design datasets, but only pertains to the visits, which are only part of the activity schedule. As for the Protocol Representation Model (PRM), it is a conceptual model that includes the study design, but has no specification details. The CDISC Study Design Model (SDM) has been developed to specify the study design. It extends the core ODM and consists of the following sub-components that model the design of the study, not its execution. The SDM is modelled in XML. Structure: epochs, arms, cells, segments, activities Workflow: decision points, branches Timing: when activities should happen 21

22 Structural Elements Structural elements define the overall structure of the study. These are illustrated in Figure 9, which describes parts of the GORD study. The full study design specification for the GORD study is given in Appendix 3: GORD-SDM.xml. Epoch: a sequential block of time for a study, e.g. the GORD study has two epochs for treatment and follow-up. Figure 4. Example of Treatment Epoch element for the GORD study Arm: defines a study arm. The GORD study has two arms for continuous and on-demand PPI. Figure 5. Example of Continuous Arm element for the GORD study CellDef: a study cell represents the intersection of an epoch and an arm. For example, the left study cell represents the treatment epoch, which occurs before randomisation into continuous or on-demand arms. 22

23 Figure 6. Example of Treatment cell for the GORD study SegmentDef: a segment has a set of activities. The continuous follow-up segment depicts the activities involved in the follow-up epoch of the continuous arm for the GORD study. Figure 7. Example of the Follow-up segment for the GORD study ActivityDef: an activity represents a point in the study at which a specific action is to be taken. There are 8 activities in the treatment segment depicted in Figure 9, such as trial start, inclusion/exclusion criteria check and randomisation (Section 2.3.4). Figure 8. Examples of the criteria check and study statistics collection activities in the GORD study 23

24 Treatment Epoch Follow Up Epoch Trial Start Study Statistics Participant Account Informed Consent Criteria Check Randomise First Visit PROM Final Visit PROM Event Visit CROM Final Visit CROM Continuous Arm First visit CROM Randomise Outcome Follow Up Segment Ondemand Arm LEGEND Columns shown as large rectangles are epochs Rows marked by arrows are arms Rectangles with bold outlines are study cells Rectangles within the study cells are segments Rectangles within the segments are activities Figure 9. Structural elements of the GORD study Workflow Elements Workflow elements specify the possible participant paths through a study. They are defined in separately from structural elements in the XML file, but they reference objects defined in the Structure section of the document [11]. The main types of workflow elements are as follows: Activities with workflow-specific roles: StudyStart, StudyFinish, PathCanFinish Entry and exit criteria for activities: EntryCriteria, ExitCriteria, Criterion Workflow paths and branching: Transition, Switch, Trigger In the following sections, we describe the key workflow elements as they are used for the GORD study. StudyStart: This identifies the single point of entry into the study for a participant, and the associated activity can be used as an anchor for other activities relative to the study start. StudyFinish: This identifies the single point at which the participant s path through the study has finished. PathCanFinish: This element allows the specification of activities for which 24

25 the workflow can provide a transition from one activity to another. Figure 10. Activities with workflow-specific roles in the GORD study Transition: This element specifies a set of branches that could be followed after an activity is completed. Transition elements are associated with activities only. A Transition references the originating activity via the SourceActivityOID attribute. Switch: Each Transition element contains one Switch element, which defines a set of TransitionDestination elements. A TransitionDestination element points to target activity and also references the ODM ConditionDef element, used for workflow transition conditions. The TransitionDefault element specifies the target activity that must take place in this transition. Figure 11. Examples of Transition and Switch elements for the eligibility criteria check activity 25

26 Trigger: Trigger elements are used to specify contingencies for unplanned events. They denote a divergence to a new path within the referenced structural element. In the example below, an alarm is triggered during the follow-up segment of the continuous arm when an event visit happens. In this transition, CROMs must be collected during this visit. Figure 12. Example of a Trigger element in the GORD study Timing Constraints Timing constraints are defined in their specific Timing section in the SDM-XML document. They determine constraints on activities and workflow transitions. Some of the core concepts of the timing constraint are now described, with reference to one example in the GORD study: The first visit PROM collection in the continuous arm should take place 1 day after the outcome of the participant randomisation is known. It is allowed to occur 1 day before or 3 days after the target time. Figure 13. Timing constraint for the first visit PROM collection in the continuous arm 26

27 Absolute and relative timing constraints: Absolute constraints limit when an activity can take place, while relative timing constraints allow the specification of the timing of two activities relative to one another. In Figure 13, the randomisation outcome and the first visit PROM collection in the continuous arm are activities relative to each other. Timing windows: An ideal time for the elapsed time between activities and workflow transitions can be specified, as well as durations before or after this ideal time. In the example (Figure 13), the pre-window duration is 1 day, while the post-window duration is 3 days. Timepoint granularity: The options available are PY, PM, PD, PTH, PTM, PTS, meaning that the activity timepoint can happen at any time in that year, month, day, hour, minute or second respectively. In the example, the timepoint granularity of PD means that the PROM collection can happen at any time in that day, and is still within 1 day after the end of the first visit, when the randomisation outcome is known to the GP. Timing types: The timing type describes the relationship of one activity relative to another, such as StartToStart, StartToFinish, FinishToStart and FinishToFinish. In the example, the timing type is FinishToStart, showing that the PROM collection (subsequent activity) should start after the randomisation outcome activity (preceding activity) is completed CDISC ODM The Operational Data Model (ODM) is a vendor-neutral, platform independent format for the interchange and archive of clinical study data [13]. The format is especially suitable when multiple systems and data sources are involved. The clinical research environment has existing clinical data management systems (CDMS) and many CDMSs already support the ODM format. In this regard, within the TRANSFoRm project, the ODM format is being used for the interchange of clinical research data. ODM is also compatible with CDISC SDTM, which is a tabular structure. The ODM models the study data as consisting of several types of entities: Item: Individual clinical data item, such as weight measurement. Item group: Closely related items are collected together into item groups. Form: A form collects a set of logically and temporally related information and represents a page in a CRF on screen or on paper. 27

28 Study event: A series of forms are collected as part of a study event. A study protocol may be designed to have a number of planned visits by study participants, and each visit can correspond to one or more study events. In the ODM XML document, the metadata entities StudyEventDef, FormDef, ItemGroupDef and ItemDef describe the types of entities that are allowed in the study. Relevant attributes that are used in the GORD study are described below and Figure 14 shows an extract from the study definition. The GORD study file is a snapshot and contains metadata: StudyOID: Clinical data entities can be identified with internal or external keys. The StudyOID uniquely identifies a study. FileType: This can be Snapshot (current state of the database with no information of the state over time) or Transactional (shows both current state and prior states of the database). Granularity: This indicates the breadth of information in the document, such as All (any type of data and metadata), Metadata (only metadata), AllClinicalData (clinical data only). Figure 14. ODM file attributes for the GORD study Metadata Elements StudyEventDef: This element is a collection of forms that form part of scheduled events (planned visits), unscheduled events (unplanned, e.g. study 28

29 termination due to serious adverse event) or common study events (e.g. adverse events). Figure 15 shows the definition for the unscheduled visits of participants in the on-demand arm, when CROMs need to be collected. The Repeating attribute indicates that this event can occur repeatedly for the participant. This study event has only one form with reference FD.EVENT_VISIT_CROM. Figure 15. StudyEventDef element for unscheduled visits in the on-demand arm FormDef: This element describes a form that is used in a study. For example, Figure 16 shows the definition of the Event Visit CROM form, which was referenced in Figure 15. This form does not occur repeatedly within the study event, as indicated by the Repeating attribute. This form has references to 13 item groups. Figure 16. FormRef element for an event visit CROM 29

30 ItemGroupDef: This element describes an item group that can occur in a study. Figure 17 shows the weight item group definition. It is a non-repeating group that has 2 items: the weight value and the weight unit, referenced by the ItemRef element. The transform:query element will be discussed in Section. It is related to how the data is extracted from the EHR source. Figure 17. ItemGroupDef element for the item group for weight ItemDef: This element describes a type of item that occurs within a study. It can have properties, such as data type, range, and codelist restrictions. Figure 18 shows the definition of the weight value item. It is a floating point number with 2 decimal points. The Question element shows the text that prompts a user to provide the data for this item. The RangeCheck element constrains the value to > 0, with an ErrorMessage element to display an error when the range check is violated. The transform:cdimbinding element at the end of ItemDef (line 15) will be described in Section Semantic extension to ODM. This links the weight item to the ontology of clinical primary care domain. 30

31 Figure 18. ItemDef element for the weight item Detailed Clinical Modelling Approach and Archetypes In the previous section, we described how clinical research studies are defined. In TRANSFoRm, the reuse of routinely collected clinical data will be achieved by prepopulating ecrfs with available clinical data directly from EHR systems. For this to be possible, the clinical data needs to be semantically interoperable between the ecrf (TRANSFoRm study system) and the EHR system. We adopt a two-level modelling approach [14] [16] to separate out the more stable domain information from the various schema implemented by the heterogeneous data sources [17]. Detailed Clinical Models (DCM) organise health information by combining knowledge, data element specification, relationships between elements, and terminology into information models that allow deployment in different technical formats [18], [19]. DCM enables semantic interoperability by formalising or standardising clinical data elements which are modelled independently of their technical implementations. The data elements and models can then be applied in various technical contexts, such as EHR, messaging, data warehouses and clinical decision support systems. Within the TRANSFoRm project, the two-level modelling approach of DCM is depicted on the first level as an information model, the Clinical Research Information Model (CRIM) [4], which defines the workflow and data requirements of the clinical research task, combined with the Clinical Data Integration Model (CDIM) [20], [21], an ontology of clinical primary care domain that captures the structural and semantic 31

32 variability of data representations across data sources. This separation of the information model from the reference ontology has been previously described by Smith and Ceusters [22]. At the second level, archetypes are used to constrain the domain concepts and specify the implementation aspects of the data elements within EHR systems or patient registries. We use the Archetype Definition Language (ADL) to define the constraints and combine them with CDIM concepts in specifying the appropriate data types and range values. The two-level modelling approach, using the concept of archetype for detailed clinical content modelling, has been adopted by ISO/CEN [23], [24]. Based on the OpenEHR framework [25], this approach makes it possible to separate specific clinical content from the software implementation. The technical design of the software is driven by the first level information model which specifies the generic information structure of the domain. The archetype defines the data elements that are required by specific application contexts e.g. different clinical studies Semantic extension to ODM A study participant s clinical data space is represented by a generic reference model. The reference model is based on TRANSFoRm Clinical Research Information Model (CRIM), which is based on the Biomedical Research Integrated Domain Group (BRIDG) model of the semantics of protocol-driven clinical research [26]. Data elements retrieved from EHR sources are considered as PerformedObservationResult class (BRIDG class). PerformedObservationResult is extended so its attribute is of type Element (openehr class). The tabular structure PerformedObservationResult Element is compatible with CDISC SDTM and ODM forms. Figure 19 shows how ODM can be semantically extended via the ItemGroupDef and ItemDef elements to be interoperable with EHR clinical data that are represented in CRIM reference model, with a binding to the CDIM ontology. 32

33 Figure 19. Link between ODM and the Reference Model via archetypes For example, in Figure 18, we mentioned the transform:cdimbinding element for form item Weight. This element is a TRANSFoRm extension to ODM, for semantic interoperability. The weight concept is identified in the CDIM ontology as concept CDIM_ This binding enables to maintain the link between the form item and the weight value after it is retrieved from the EHR system for pre-population. In Figure 17, the last element in the item group definition for Weight is a transform:query element, which is also a TRANSFoRm extension to ODM: This query ID refers to a data extraction request for weight value, unit and time of measurement, as shown in Figure 20, indicated by the three CdimConcept elements. 33

34 Figure 20. Part of the data extraction query for weight The request for data extraction of the three weight-related data items is accompanied by constraints that specify how the data should be filtered. These are specified in the Archetype section of the query, as shown in Figure 21 (definition), Figure 22 (ontology term definitions) and Figure 23 (ontology term bindings). Figure 21. Archetype for weight - definition 34

35 Figure 22. Archetype for weight - ontology definitions Figure 23. Archetype for weight - ontology term bindings Archetype-based Query Model for Eligible Patient Identification and Patient Data Extraction The archetype approach establishes the semantic foundation for describing clinical data elements, and enables the semantic interoperability across different EHR systems. Based on the archetypes, a query model is developed to facilitate query formulation for eligible patient identification and patient data extraction. 35

36 As described in Figure 24, the query model allows formulating three different types of query requests and their responses, including EligibleSubjectCountRequest, requesting for counts of eligible subjects. Its response returns the count of found eligible subjects. FlagEligibleSubjectRequest, flagging eligible subjects. Its response returns the count of flagged subjects. DataExtractionRequest, retrieving data from the source for the flagged subject(s). Its response returns the requested data. All these query requests can embed query criteria. The query criteria allow using archetypes for clinical data specifications, and allow doing arithmetic calculations and calling functions (Figure 25). Furthermore, complex Boolean logic and temporal constraints can be constructed in the query criteria. The actual queries formulated by the model are encoded in xml format. Figure 20 - Figure 23 describe an example of DataExtractionRequest which requests for the patient weight using the TRANSFoRm archetype weight as the data element specification. 36

37 Figure 24. Archetype-based query model main elements 37

38 Figure 25. Query model - expressions 38

39 2.3 GORD Evaluation Study Specification An instance specification for the GORD evaluation study is developed by applying the modelling methodology and all the models described in previous sections. The study is encoded in CDISC SDM with a number of TRANSFoRm extensions to cover the missing functions such as form pre-population, randomisation, mobile interface rendering, and alarm symptom monitoring. In the GORD SDM XML, all the TRANSFoRm extensions are defined under the namespace transform and are denoted in the form of <transform:xxx> Study Workflow The GORD evaluation study is a single-blind, multi-centre, parallel-arm, cluster randomised controlled trial to assess the effectiveness of the TRANSFoRm tools in recruiting patients into a study in primary care and obtaining complete case data. The complete workflow is depicted using a flowchart in Figure 26. The study has two arms: on-demand or continuous prescribed use of PPI. Participants and clinicians will not be blinded as it is the dosage regimen that differs between groups and both patients and clinicians will see the prescribed dosage. However, the analysis of the study will be blinded. The study will span 32 weeks - 24 for the recruitment of participants and 8 weeks for follow-up. Following CDISC SDM, the study is designed to comprise two epochs (Figure 4): Treatment epoch participant is randomised and PPI is prescribed. Follow-up epoch participant is followed up with PROMs and CROMs. The intersection of the two epochs with the two arms produces three cells (Figure 6), each of which contains one segment: Treatment cell represents the part of study (i.e. treatment epoch) with only one path, which is same for both arms. Treatment cell contains the treatment segment. On-demand follow-up cell represents the follow-up epoch for the on-demand prescribed use of PPI arm and contains the on-demand follow-up segment. 39

40 Continuous follow-up cell represents the follow-up epoch for the continuous prescribed use of PPI arm and contains the continuous follow-up segment. Figure 26. GORD evaluation study workflow Figure 9 presents the full picture of the study structure, composed of epochs, arms, cells and segments. Examples are presented in Figure 4 - Figure 7. Segments are the basic building blocks of study design. A segment usually specifies a combination 40

41 of planned observations and interventions, which may or may not involve treatment, during a period of time. A segment includes one or more activities. In the design specification of the GORD study, the treatment segment contains all the activities that will be carried out during the patient s first visit (visit 1 week 0) to the primary health care centre (PHC), including: Trial start: the patient will be flagged as a potentially eligible subject if the patient presents related diagnosis, symptoms or prescriptions. The trial start activity embeds a flagging patient query which will trigger the study when the patient is flagged. Check inclusion/exclusion criteria: after the patient is flagged and the study is triggered, the GP is asked to check the inclusion and exclusion criteria. Collect study statistics: if the patient is not eligible, anonymized information of age and gender is saved as study statistics. Informed consent: if the patient is eligible, the GP gives the patient the study information. Randomisation: if the patient consents, the GP confirms the age and gender. This information is used to allocate patients to randomisation blocks in the randomization. After completing this point, the patient is randomized (but the outcome is hidden from the GP until the end of the consultation when the dose is prescribed to avoid biasing the GP). Create study account for the patient: at this point an ID, user name and password is created for the study participant. First visit CROM: the GP checks the prefilled information and adds missing information in the CROM. Randomisation outcome: when the CROMs are completed, the GP will be informed of the randomization outcome and can prescribe the appropriate dose. The participant will receive information about when and how to fill out the PROMs. The GP/PHC books a follow up appointment with the participants 8 weeks later. After this point, the study enters the follow-up period and is running in the two parallel arms. The on-demand follow-up segment contains the activities that will be carried out during the follow-up period for the on-demand arm, which include: First visit on-demand PROM: The PROMs can be filled out on the patient s 41

42 smart phone or on a computer with Internet access and should be completed within 3 days of visit 1. The patient will receive reminders 2 days, 3 days and 4 days after visit 1. Final visit on-demand CROM: when the participant returns for the follow-up visit (visit 2) after 8 weeks, the CROMs are activated and the GP checks out prefilled fields and completes any missing information. Final visit on-demand PROM: the participant completes the PROMs within 3 days after visit 2. Similarly, the patient will receive reminders 2 days, 3 days and 4 days after visit 2. Event visit on-demand CROM: if the participant visits the PHC between visit 1 and visit 2, CROMs for event visits are activated. The recording of event visits between visit 1 and visit 2 is to make sure that the number of PHC contacts doesn t differ between the treatment regimes. The continuous follow-up segment contains similar activities to those of the ondemand follow-up segment, whereas the activities are specifically associated with the continuous arm. All the activities described above are defined in the study SDM XML as <sdm:activitydef >. One example has been presented in Figure 8. The execution order between these activities is defined in the SDM workflow section <sdm:workflow>. One example is presented in Figure 11. The timing constraints are defined in the SDM timing section <sdm:timing>. One example has been presented in Figure 13. Note that SDM XML allows only explicit timing constraints between activities not study events. The 8-week constraint between visit 1 and 2 is therefore defined between the last activity of visit 1 (Randomisation Outcome) and the first activity of visit 2 (Final Visit CROM) for either arm Case Report Forms The GORD study data consists of two parts: care-reported outcome measures (CROMs) and patient-reported outcome measures (PROMs). CROM represents the category of all types of objectively reported care outcomes, collected at the physician s office including data from the patient s electronic health record, such as BMI, blood pressure, laboratory data. PROM represents the category of all types of subjective reported outcomes such as health-related quality of life, treatment 42

43 satisfaction, subjective health status and subjective symptoms. All study parameters (CROMs, PROMs) are assessed with electronic questionnaires. No invasive procedures, physiological investigations, clinical or laboratory tests will be performed as part of this study. GPs report care reported outcome measures (including age, gender, BMI, smoking, PPI consumption and medical history) at the time of inclusion, in case of events and at the 8 week follow-up. Participants complete patient reported outcome measures (including demographics, GORD symptoms, health related quality of life, self-rated health and use of GORD medication) through mobile phone or web at the time of inclusion and after 8 weeks. Appendix 2 includes paper versions CRF for both CROM and PROM. Based on the study plan and paper versions CROM and PROM, the SDM XML encoded CROM forms for visit 1, visit 2 and event visit, PROM forms for the ondemand arm visit 1 & 2, and PROM forms for the continuous arm visit 1 & 2. CROM forms are all same for both arms, whereas PROM forms are different between the two arms. CROMs will be deployed into an EHR system and triggered from GP desktop, while PROMs can be accessed through mobile phone or web form and can be used by the patient at any location (e.g. from home). Because CROMs and PROMs are technically managed and rendered by different software components, a TRANSFoRm extension <transform:formtype> is introduced into <FormDef> to indicate how this form should be processed. Examples of <FormDef> and its internal elements are presented in Figure 16 - Figure 18. In addition to the CROM and PROM forms, the SDM XML includes a number of other forms as they are used in various study activities, including inclusion/exclusion criteria form, study statistics form, randomisation form to capture input to the randomisation service, and randomisation outcome form to inform GPs about the randomisation result Flagging Patients and Form Pre-population When a patient visits the primary care centre for any complaint, the patient will be flagged by the system as a potentially eligible patient for the GORD study if 1. There is a GORD diagnosis or a diagnosis of symptom of GORD i.e. heartburn or acid regurgitation already in the EHR 43

44 OR 2. The GP enters a GORD diagnosis or diagnosis of symptom of GORD i.e. heartburn or acid regurgitation during the consultation OR 3. A PPI has been prescribed or is prescribed. AND The patient is aged between 18 and 65 years. These criteria are formulated in a FlagEligibleSubjectRequest query. This flagging query is embedded in the Trial Start activity and will trigger the execution of the GORD study once the patient is flagged. As described in section 2.3.2, CROM forms are deployed and triggered within an EHR system. When a CROM form is triggered, the form is either presented in a stand-alone application on the GP s desktop, such as a web browser, or embedded with the EHR user interface. Certain data items can be retrieved from the GP s EHR system, and can be used to pre-populate related questions in the form. Those prepopulated fields however have to be manually checked (and potentially edited) by the GP before the form is submitted. The following table Table 1) summarises the data elements which are expected to be available from GP s EHR system and so can be used to pre-populate relevant GORD CROM forms. Table 1 lists the data elements, their archetypes and the visits (marked by X ) at which they will be collected. Some data items are collected only at the time of first visit, while some others have to be collected every time. The data elements are listed together with their identifying terminology codes, where prescriptions use ATC codes and diagnosis use ICD-10 codes. Data extraction queries for retrieving these data elements are formulated as DataExtractionQueryRequest for each of them. All the queries are defined in the <transform:queries> section in the SDM XML, and are linked to the corresponding <ItemGroupDef> elements in the related ODM forms through UUID, following the approach elaborated in Section Semantic extension to ODM. Figure 16 - Figure 23 have presented the CROM question, the query, and their linkage, using the data element weight as an example. 44

45 Table 1. EHR data elements for CROM form pre-population Data Element Archetype Visit 1 Visit 2 Event Visit Birth year (YYYY) Date of Birth X Gender Gender X Height Height X Weight Weight X X X PPI use (A02BC) Prescription X Antacids (A02A)/alginate Prescription X X X (A02BX13) Esophagitis (K20) Diagnosis X X X Barrett s oesophagus (K22.7) Diagnosis X X Randomisation Study participants will be randomized to either on-demand or continuous use of PPI for 8 weeks. Patients randomized to on-demand use will be prescribed PPI in the presence of symptoms but no more than 40 mg omeprazole per day. Patients randomized to continuous use of PPI will be prescribed 20 mg omeprazole per day. This will be a block randomization containing 4 blocks based on age (50 years and below, and above 50 years) and gender. The study will not be blinded as it is the dosage regimen that differs between groups so there won t be a need to break the randomization code. The input parameters to the block randomisation function (i.e. age and gender) are captured by the Randomisation Form which will be triggered by the randomisation activity at patient s first visit. Collected data are sent to the TRANSFoRm Study System which assigns the patient to a block and applies the randomisation function. The outcome is hidden from the GP until the end of the consultation when the dose is prescribed. The outcome is then informed through the Randomisation Outcome Form. The TRANSFoRm randomisation function requires proper configuration of the blocking variables in the study definition. In the case of the GORD study, the blocking 45

46 variables are year of birth and gender. Because the current version of SDM XML does not cover randomisation design, the blocking variables have to be specified in a TRANSFoRm extension element <transform:participantsegmentvariables>. As shown in Figure 27, the variables year of birth and gender are linked to the corresponding fields in the Randomisation Form through the ItemOID, and their range definitions specify the group arrangement. Another parameter required by the randomisation function is the estimated number of recruited participants. The number is used to initialise the random allocation sequence. This parameter is defined as a <sdm:parameter> in the <sdm:summary> section (Figure 28). Figure 27. Randomisation blocking definition 46

47 Figure 28. Study parameter planned number of recruited subjects 2.4 Conclusion In Section 2. Specification of a functional ecrf to be integrated with an EHR, we described how a functional ecrf is made semantically interoperable with an EHR system to enable the pre-population of ecrfs with available primary clinical data. A two-level modelling approach has been adopted, with the first level being guided by the reference model for the clinical research domain (CRIM), and on the second level, archetypes are used to constrain the reference model. Clinical data elements have a binding to the CDIM ontology that describes the primary care clinical domain. To maintain the semantic meaning when using ecrfs, the CDISC ODM standard has been extended to allow for archetypes to further define form elements and queries to EHR systems to extract specific clinical data items. 47

48 2.5 References [1] TRANSFoRm Consortium, TRANSFoRm Annex I - Description of Work, Version Nov [2] World Health Organization, International Classification of Primary Care, Second edition (ICPC-2), WHO. [Online]. Available: [Accessed: 21-May- 2014]. [3] WHO Collaborating Centre for Drug Statistics Methodology, ATC Structure and principles. [Online]. Available: [Accessed: 21-May-2014]. [4] W. Kuchinke, T. Karakoyun, and C. Ohmann, Clinical Research Information Model, Project Deliverable TRANSFoRm Deliverable D6.2, V1.0, May [5] Clinical Data Interchange Standards Consortium, CDISC. [Online]. Available: [Accessed: 21-May-2014]. [6] P. Leysen, H. Bastiaens, P. van Royen, L. Agreus, and A. N. Andreasson, D1.1 Detailed Use Cases, TRANSFoRm Project Deliverable, Feb [7] A. N. Andreasson, R. A. Verheij, and K. Hek, The effect of on demand versus continuous use of proton pump inhibitors on reflux symptoms, quality of life and self-rated health in patients with gastro-oesophageal reflux disease, Evaluation Study Research Ethics Protocol Protocol ID KIANANDR (draft), Apr [8] N. Mastellos, K. Huckvale, M. Larsen, J. Car, A. N. Andreasson, and L. Agreus, Protocol for evaluation studies for the GORD use case. Mar [9] Clinical Data Interchange Standards Consortium (CDISC), CDISC Foundation Standards. [Online]. Available: [Accessed: 23-Apr-2014]. [10] Clinical Data Interchange Standards Consortium (CDISC), Representation Model for Planning and Design of Medical Research Protocol. [Online]. Available: [Accessed: 23-Apr-2014]. [11] Clinical Data Interchange Standards Consortium (CDISC), CDISC Study Design Model in XML (SDM-XML) Version 1.0, [12] Clinical Data Interchange Standards Consortium (CDISC), Study/Trial Design Model: Study Design in XML Version 1.0. [Online]. Available: [Accessed: 23-Apr-2014]. [13] Clinical Data Interchange Standards Consortium (CDISC), Operational Data Model. [Online]. Available: [Accessed: 24-Apr-2014]. [14] A. L. Rector, W. A. Nowlan, S. Kay, C. A. Goble, and T. J. Howkins, A framework for modelling the electronic medical record, Methods Inf. Med., vol. 32, no. 2, pp , Apr [15] S. B. Johnson, Generic data modeling for clinical repositories., J. Am. Med. Inform. Assoc., vol. 3, no. 5, pp , [16] T. Beale, Archetypes: Constraint-based domain models for future-proof information systems, [Online]. Available: 48

49 _oopsla_2002.pdf. [17] S. N. Lim Choi Keung, L. Zhao, J. Rossiter, M. McGilchrist, F. Culross, J.-F. Ethier, A. Burgun, R. Verheij, N. Khan, A. Taweel, V. Curcin, B. Delaney, and T. N. Arvanitis, Detailed Clinical Modelling Approach to Data Extraction from Heterogeneous Data Sources for Clinical Research, in 2014 Summit on Clinical Research Informatics, San Francisco, USA, [18] W. Goossen, A. Goossen-Baremans, and M. van der Zel, Detailed Clinical Models: A Review, Healthc. Inform. Res., vol. 16, no. 4, pp , Dec [19] W. T. F. Goossen and A. Goossen-Baremans, Bridging the HL7 template archetype gap with detailed clinical models, Stud. Health Technol. Inform., vol. 160, no. Pt 2, pp , [20] J.-F. Ethier, O. Dameron, V. Curcin, M. M. McGilchrist, R. A. Verheij, T. N. Arvanitis, A. Taweel, B. C. Delaney, and A. Burgun, A unified structural/terminological interoperability framework based on LexEVS: application to TRANSFoRm, J. Am. Med. Inform. Assoc., Apr [21] J.-F. Ethier, M. M. McGilchrist, A. Burgun, and F. Sullivan, D6.3 Data Integration Models, TRANSFoRm Deliverable, May [22] B. Smith and W. Ceusters, HL7 RIM: an incoherent standard, Stud. Health Technol. Inform., vol. 124, pp , [23] EN Association, The CEN/ISO EN13606 standard. [Online]. Available: [Accessed: 25-Apr-2014]. [24] P. Muñoz, J. D. Trigo, I. Martínez, A. Muñoz, J. Escayola, and J. García, The ISO/EN Standard for the Interoperable Exchange of Electronic Health Records, J. Healthc. Eng., vol. 2, no. 1, pp. 1 24, [25] openehr Foundation, openehr. [Online]. Available: [Accessed: 25-Apr-2014]. [26] BRIDG Founding Stakeholders, BRIDG Model. [Online]. Available: [Accessed: 25-Apr-2014]. 49

50 Appendix 1: User Requirements Elicitation (manuscript) Eliciting user decision requirements for designing computerised diagnostic support for GPs Talya Porat, Amanda Woolley, Olga Kostopoulou Department of Primary Care and Public Health Sciences, School of Medicine, King s College London Correspondence: Olga Kostopoulou, King s College London, School of Medicine, Department of Primary Care and Public Health Sciences, Capital House, 42 Weston Street, London SE1 3QD, UK olga.kostopoulou@kcl.ac.uk Tel: +44 (0) Fax: +44 (0)

51 ABSTRACT Background: Despite its 40-year history, computerised diagnostic support is not commonly used in medicine. Aim: To identify user decision requirements for the design of a computerised diagnostic support tool for GPs. Design and Setting We employed the decision-centred design framework of cognitive systems engineering. All data were elicited from GPs in London and pertained to consultations with patients, either real or simulated. Methods: To collect background information about the domain, we conducted in situ observations and interviews with 8 GPs and performed a hierarchical task analysis of the diagnostic task. To identify key components of expert decision making, we used cognitive task analysis on 58 existing verbal protocols: 17 GPs thinking aloud while seeking information to diagnose cases on computer and 18 GPs describing cases that they diagnosed intuitively. Results: To support the decision requirements of the diagnostic process that we identified, we propose six solutions for the design of computerised tools: (1) Suggest possible diagnoses to consider early on in the consultation before hypothesis testing starts. (2) Provide checklist of important, diagnostic symptoms and signs for the suggested diagnoses. (3) Make salient important information in the patient s record. (4) Suggest examinations and investigations. (5) Alert when GPs have not checked for serious or urgent diagnoses. (6) Facilitate coding and data entry. Conclusion: Studies employing multiple human factors techniques and data types are rare. Our approach enabled us to propose interface design solutions that go beyond existing differential diagnosis generators, aiming to improve acceptance of the resulting tool by GPs. Keywords: General Practice; Clinical Decision Support Systems; Decision making; Diagnostic Errors 51

52 HOW THIS FITS IN We provide a new perspective on the design of diagnostic support beyond the wellknown differential-diagnosis generators. We emphasise the need for early and interactive support, coupled with facilitated data entry and coding throughout the consultation. The support tool should make the diagnostic task more efficient and less error-prone and should reduce the GPs burden of coding. This would result in a richer and easily searchable health record and increase the tool s perceived benefits and likelihood of use by GPs in their everyday practice. 52

53 INTRODUCTION Diagnostic errors are common, harmful to patients and costly. They are the cause of preventable mortality and morbidity and the commonest cause of litigation against GPs in both the UK and USA. 1 2 Computerised diagnostic support has been one of the strategies proposed for the reduction of diagnostic error. 3 However, more than 40 years after De Dombal s AAPHelp, 4 the use of computerised diagnostic support is not widespread. One large roadblock to acceptance of diagnostic support is lack of integration with the physician s workflow and the electronic patient record (ehr). 5-7 Commercial diagnostic tools, such as DXplain, Isabel, and SimulConsult, are stand-alone applications, requiring physicians to switch from their ehr to the tool and add information twice, costing precious time. Moreover, physicians must first decide to consult the system, though they may not recognise when they need advice. 8 Finally, these systems are designed to be used after the physician has collected as much information as possible from the patient, which the system will need in order to provide diagnostic suggestions ( differential diagnosis generators ). Thus, advice comes late in the diagnostic process, and the information that physicians enter into the system depends on the initial hypotheses considered. The system s advice, and thus its potential value, depends on how users can convey to the DSS their personal understanding of a case by selectively entering clinical findings. 9 A recent experimental study found that presenting GPs with differential diagnoses to consider early on in the consultation, before they start testing hypotheses, significantly improved diagnostic accuracy over control. 10 Taking all this evidence into account, we aimed to elicit decision requirements for the design of a diagnostic tool for GPs that is being developed as part of the TRANSFoRm project (transformproject.eu). To elicit requirements, we employed a multi-method approach encompassing a range of human factors techniques and data sources. Below, we present the methods and associated results, while the table of user requirements and associated design solutions is presented in Appendix 1. 53

54 METHODS AND ASSOCIATED RESULTS We employed a decision-centred design (DCD) framework of cognitive systems engineering that advocates focusing on difficult key decisions and non-routine situations, where errors may lead to injury and/or death. 11 It is therefore suitable for designing support for medical diagnosis. DCD uses cognitive task analysis (CTA) techniques to identify the key decisions. 12 It then translates these into requirements that guide the design of solutions for supporting decision making in challenging situations, assuming that the routine requirements will be incorporated along the way. The DCD framework includes five stages. Here, we describe the first three stages: preparation, knowledge elicitation, and analysis and representation. Stage 1: Preparation The preparation stage seeks to gather background material about the domain that will inform about the nature and range of the tasks involved and identify cognitively complex task elements. Preparation started with reviewing an existing hierarchical task analysis (HTA) of the primary care consultation. 13 HTA models tasks as hierarchies of goals and sub-goals, with plans that show how sub-goals should be carried out. 14 We aimed to refine the parts of the HTA that related to diagnosis. For this purpose, we observed 8 GPs (5 male, mean 8.6 years in general practice, SD 6) consulting with their patients and offered them recompense at the standard clinical hourly rates. The researcher (TP) sat in the consulting room and unobtrusively observed the clinical interaction, taking notes. A total of 104 clinical encounters (23.5 hours) were observed. The researcher noted how the GPs used their ehr, for which tasks and at which stage in the workflow. Notes from all observations were compared to gain a better understanding of the information requirements of the diagnostic task. GPs were then interviewed about their diagnostic strategies in the cases observed earlier, past diagnostic errors and suggestions how diagnostic support could help avoid these errors. On the basis of the observations and follow-up interviews, we refined and expanded the diagnostic component of the HTA (Figure 1). 54

55 0. Provide consultation with GP PLAN 0: Before starting surgery do 1, 2, 3 and 4 in any order and combination. If patient has arrived and when ready to receive patient - 5. When patient comes into the consultation room If do not know or cannot remember patient s information go back to 4. Throughout the consultation do 11. If decided on course of action do 9 and 10. If further information regarding present visit needs to be obtained 7. If next patient arrived and when ready to receive patient 5 and carry on as for previous patient. 1. Review the list of appointments 2. Check whether first patient has arrived (system indication) 3. Open patient health record 4. Get familiar with patient s clinical history 5. Call patient in 6. Establish why patient has come 7. Obtain new information PLAN 7: Do 7.1. As appropriate and if available 7.2 and 7.3, in any order and combination. 7.1 Obtain new information from patient PLAN 7.1 Do any to in any order and combination, as appropriate Obtain information verbally Obtain information via observation Obtain information via physical examination Obtain information via near-patient testing (e.g., dipstick urine analysis) 7.2 Review relevant pathology results 7.3 Review relevant clinical correspondence 8. Decide on course of action 9. Safety net 10. Carry out next steps 11. Record consultation Figure 1: An extract from the refined HTA about the clinical consultation, with associated plans and goals. For illustration purposes, only goal 7.1 is re-described. Stage 2: Knowledge elicitation The knowledge elicitation stage uses cognitive task analysis (CTA) methods to elicit critical incidents and key components of expert decision making. For this purpose, we analysed existing think-aloud protocols of GPs diagnosing patient cases presented on computer and interview protocols of GPs describing past cases of intuitive 55

56 diagnoses. 1. Think-aloud protocols We used data from an ongoing study by the last author that investigates how GPs deal with early presentations of cancer (lung, colorectal, and myeloma). Participating GPs viewed a series of patient cases on computer. After some initial information about the patient and the reason for encounter, GPs requested further information in order to diagnose. They could take a history and request results of physical examinations and laboratory tests. A researcher provided responses from a predetermined list. Furthermore, participants were instructed to think out loud. 15 We used the first 34 think-aloud protocols from this study: 11 pertained to lung cancer, 11 to myeloma and 12 to colorectal cancer. We used the situation assessment record (SAR) method to analyse the protocols. 16 In SAR, the timeline for an event specifies the points at which the expert engaged in situation assessment and decision making. Each of the events may be defined in terms of cues (information items), expectations, hypotheses, strategies, goals and decision points (See Table 1 for an excerpt from the SAR analysis. The scenario features the first visit of a patient with early myeloma). TABLE 2: Excerpt from the SAR analysis of a think-aloud protocol of GP 9 (cancer diagnosis study) Situation Assessment 1 Cues Expectations Goal 67, female, a bit on the heavy side, has a history of blood pressure, hypertension and arthritis. She s on medication for her blood pressure and she s been seen with a back problem relatively recently. Referred for physio and note that she attends quite frequently. Seems to be well but is holding her back and does seem to be in pain. So she s talking about having back pain. She s taken cocodemol and the pain hasn t got better. I m assuming that she s wanting better pain relief. She is already taking co-codamol, she s had the physio. Ask about the physio / explore the reason for encounter Decision point 1 Situation Assessment 2 Cues Hypotheses 1 and 2 Did you feel that the physiotherapy made any difference or whether the treatment is still ongoing? I've seen the physio twice now. She recommended some exercises to do at home. I try to do them every day. They keep me active but I am not sure how much they are helping. I have some follow up sessions booked. So it may be that she needs to be encouraged to be a little bit more patient or I m thinking about disc prolapse Goal Differentiate between hypotheses 1 and 2 56

57 Decision point 2 Situation Assessment 3 Cues Expectations Hypothesis 2 Goal Decision point 3 Situation Assessment 4 Cues Hypothesis 2 Goal Decision point 4 Has the pain got any worse? After the pain started it has been the same constant aching most days. I think she s only had two sessions of the physio, she s only been doing the exercises at home for a little while. I m thinking she probably just needs to be a bit more patient. Mechanical pain ( So it does sound like it s mechanical low back pain ). Increase medication to control the pain Ask her how much of the co-codemol she s taking and if she s getting any problems with it such as constipation. I have been taking the co-codamol tablets, as I was told, mostly 8 per day. They take the edge off the pain but it doesn't go completely. Mechanical low back pain She s taking up to the maximum dose so I don t particularly want to change anything there. She should continue with her exercises and give the physio a bit more chance, so I ll ask her to come back if it doesn t settle and to continue with her physio. 2. Interview protocols based on the Critical Decision Method We also used data from a completed study by the second and third authors, where GPs were interviewed about patient cases that they believed to have diagnosed by intuition. 17 We used these data in order to gain an understanding of cognitive processes during intuitive decision making in addition to the more analytical reasoning approach of the think-aloud protocols. At the interviews, the researchers prompted the GPs systematically, following the Critical Decision Method (CDM) that has been used in numerous domains to investigate the cognitive components of proficient performance. 18 They thus collected 24 CDM protocols from 18 GPs, which we re-analysed using SAR, to enable comparison with the analysis of the think-aloud protocols (See Table 2 for an excerpt from the SAR analysis. The scenario features a patient with ovarian cancer). TABLE 3: Excerpt from the SAR analysis of a CDM protocol of GP 8 (intuition study) Situation Assessment 1 Cues 67, female, not happy with previous consultation, felt unwell, tired with no energy, dizzy, loss of appetite, weight loss (a stone over a year), overweight, ex-smoker, previous tests were all normal. Non-diagnostic cues: A very, very frequent attender and has multiple social and psychological problems. And she wears you out 57

58 Expectations Decision point 1 Situation Assessment 2 Cues Expectations Hypothesis 1 Goal Decision point 2 Situation Assessment 3 Cues Expectations Hypothesis 2 Goal Decision point 3 She lost a stone in weight over the last year, less concerning since the weight loss was over a long period of time ( but if you lost that sort of weight over a month or two or three, then you re going to be worried ), in addition, she is a biggish lady, and therefore a stone didn t feel that much in that sense. But she d never lost weight before. And not many 67 year olds go on a diet. I mean, young women and men do, but not a 67 year old. Perform a full examination Abdomen normal, chest normal (full examination normal) I think my guts told me that there wasn t anything too much wrong with this lady Seemingly nothing is wrong, possibly irritated bowel Confirm nothing is wrong with blood tests Perform screen tests (to look at all options): blood tests (ESR, kidney and liver, sugar, thyroid, FBC) and chest x- ray. Blood tests normal, chest x-ray normal Given problem with previous consultation best to follow through to avoid upset: I did pick up a feeling in her voice that she was unhappy with the previous consultation with my partners, and I thought by following her through and making sure that nothing went wrong, as it were, that we might avoid any complaint or upset. So that was really why I brought her back. Nothing serious Arrange a follow-up Arrange an appointment in two weeks Stage 3: Analysis and representation The HTA and the analyses of these different data types enabled us to elicit cognitive requirements (Table 3) for which we propose user interface design solutions. The proposed solutions are summarised in a decision requirements table (Appendix A), as suggested by the DCD framework. TABLE 4: Cognitive requirements of the diagnostic process 1 Retrieval and integration of information 1.1 Retrieval of relevant and important information from the health record 1.2 Integration of the patient s reason for encounter with relevant information from the health record 2 Diagnostic hypotheses generation 3 Hypothesis testing and refinement (including decisions about what and how much information to elicit) 3.1 What questions to ask the patient 3.2 If and what examinations to perform 3.3 If and what investigations to perform or order 58

59 3.4 How to interpret the information elicited 3.5 When to stop eliciting information (define a stopping rule) 4 Deciding on a course of action 1. Retrieval and integration of information GPs must retrieve and integrate information from different sources to build an understanding of the patient s condition. They scan the patient s health record looking for significant information, either by browsing in a structured way (problems, medications, previous encounters) or by actively filtering information (display only high priority problems). Important information may be missed, if it is not wellpresented and emphasised in the record or due to time constraints and distractions: I missed once a cancer case. It was a woman in her 50s coming with a headache, I missed the information that when she was young she had cancer, it was in the record but at the very bottom, I didn t scroll down (GP3, post-observation interview). In addition, GPs may have preliminary strong assumptions from reading information in the ehr, even before starting the consultation: I thought, he s going to be anxious, I almost, you know, if you d said, What s this man going to come in with? I would say, Anxiety, definitely. Because I mean, the last four times it had been about anxiety (GP 5, CDM protocols, the patient was later diagnosed with heart disease). Errors in initial information integration of the patient s reason for encounter with information from the health record can arise if the GP is guided by strong assumptions: Well, in terms of, she s obviously attending frequently, she s a hypertensive, she s mildly obese She s got chronic back problems, probably osteorelated and probably associated with her extra weight So my advice she needs to do a bit more herself, perhaps get more fitter with activities (GP 6, think-aloud protocols, early presentation of myeloma). Interface design solution A diagnostic tool should make salient important and relevant patient information in the record, e.g., risk factors, relevant family history, and chronic diseases, thereby increasing the GP s situation awareness Diagnostic hypotheses generation GPs generate a small number of diagnostic hypotheses early in the consultation, which guide the diagnostic process and can determine its outcome. 20 A pre-existing 59

60 disease, a colleague s opinion, situational factors (e.g., time of the year or a previous patient), and assumptions about the patient (e.g., frequent consulter) may install a leading hypothesis at the exclusion of other alternatives. I think I was agreeing with the earlier doctor, who saw her a week earlier, that maybe it was a sprain I think it was that feeling of she comes here often, and she s quite anxious because her husband left her recently and she was all alone and she s struggling. And she wants reassurance that everything is doing okay (GP 1, CDM protocols, missed foot fracture). Interface design solution A diagnostic tool should display a list of possible diagnoses by integrating information from the ehr with the reason for encounter. This aims to reduce narrow focus on one diagnosis developing early in the encounter. The following information should be taken into account: Demographic data (age, gender, etc.) Risk factors (smoking, alcohol, family history, BMI, etc.) Chronic diseases (e.g., asthma, diabetes) Recent examination and investigation results (e.g., last 3-6 months) Following the results of a recent experimental study 15, the tool will display the list of possible diagnoses as soon as the GP enters a reason for encounter. This aims to expand the hypothesis space and enable GPs to consider other possibilities, before sticking to a leading hypothesis at the exclusion of others. 3. Hypothesis testing As the problem space is large, information gathering would be endless, was it not hypothesis-driven and guided by a stopping rule. 21 In addition to deciding what questions to ask, examinations to perform and investigations to order, GPs have to decide when to stop gathering information and proceed to diagnosis and/or management. Physicians seeing the same patient can differ greatly in their diagnostic approach, as illustrated in the following example from the think-aloud protocols. At the first patient presentation, GP3 asked the patient 18 questions, performed 4 examinations and ordered 8 investigations before deciding to refer the patient to hospital: So it s becoming a bit more, looking like this lady may unfortunately have myeloma, which would fit with this persistent worsening back 60

61 pain, mild anaemia and raised globulins and urine proteins. So I m referring her to the haematologist and I m going to ask as an urgent... GP9 seeing the same patient asked only 3 questions and told her to come back if the back pain persisted. At the second patient presentation (with prolonged and worsening symptoms), GP9 ordered a single investigation (X-ray of the back) and upon discovering that it was normal, the GP decided to prescribe pain relief: Am I concerned that there s something that we re missing or should I just try her for a bit longer with better analgesia? So I think, given that we haven t tried better analgesia, I think that s the next thing that I would do. So I d stick with my diagnosis for now and increase her analgesia. This example illustrates two factors in the diagnostic process that may lead to error: first, that asking too few questions (presumably driven by a single hypothesis) may lead to misdiagnosis, as important information will not be discovered. Second, that once the physician adopts an interpretation, it may prove resistant to change, despite discovering new information that is inconsistent with that interpretation. Interface design solution In addition to presenting a list of diagnostic suggestions, the tool should enable users to click on a suggested diagnosis and view the important features (symptoms and signs) that can change the likelihood of the diagnosis. Users can check for these features in the patient and tick either Yes or No to indicate their presence or absence. The ehr will be automatically updated and so will the list of suggested diagnose, if appropriate. For example, the order of the diagnoses may change according to their updated likelihood, and diagnoses may be added or removed. The tool should also propose examinations and investigations that could differentiate between the suggested diagnoses. In this way, GPs are likely to elicit and consider more information. Finally, to avoid missing life-threatening or urgent conditions, the tool should alert GPs who conclude the diagnostic process without checking any of the symptoms and signs of these conditions. 4. Decide on course of action Errors in management decisions can stem directly from misdiagnosis. They can also occur if the physician does not safety net for serious possibilities and only manages for what he/she considers to be the most likely diagnosis. Finally, errors may occur 61

62 due to insufficient knowledge about the most appropriate way to manage a specific disease. Interface design solution When the GP enters a diagnosis, he/she should be able to view guidelines with direct links to forms for referral, investigations and prescribing, in relation to that diagnosis. DISCUSSION Summary: Using multiple methods and types of data, we elicited cognitive requirements of the diagnostic task and proposed specific user interface design solutions for a diagnostic tool. The tool should adhere to known design requirements for workflow and ehr integration and suggest diagnoses for GPs to consider early in the process, so that narrow focus on a single hypothesis is lessened. The tool should have at its heart the reason for encounter and should facilitate data coding and insertion. It should suggest symptoms and signs that are important for the relevant diagnoses. It should highlight significant information in the ehr and alert GPs if they have not checked for life-threatening or urgent conditions. These features could enhance the tool s usefulness and acceptability. Strengths and limitations: The strength of our work resides in its use of multiple methods of data collection and analysis (observations, interviews, HTA, CTA, and SAR) to elicit cognitive requirements of the diagnostic task. This has not been done before for the development of diagnostic support. Another point of strength is our use of different data types that covered the spectrum of diagnostic reasoning from the intuitive to the analytical. This ensures that the requirements elicited and design solutions proposed are relevant to and can support the different modes of clinical thinking The use of multiple data sources is novel in the requirements elicitations literature. A limitation of this study is by focusing on the diagnostic task and tailoring design solutions to the diagnostic tool, we disregarded the interaction of the diagnostic task with other non-diagnostic tasks performed by the GP during consultation (e.g., prescribing medication). This could have an effect on the overall user experience with the diagnostic tool, and should be evaluated. 62

63 Comparison with existing literature: We are not aware of any diagnostic support systems for healthcare that were designed on the basis of a detailed requirements elicitation process, which may partly explain their lack of widespread adoption. Implications for research and/or practice: The RfE is the trigger of the proposed tool. Entering a RfE at the start of the consultation may require GPs to change the way that they interact with their computer. For some, this change would be relatively small. However, GPs differ greatly in the timing and amount of information that they record; some were observed to record only the consultation outcome and only after the patient has left the room. For these GPs, using the tool would be a challenge. Another potential problem is the multiple RfEs that patients often present with, in which case the GPs will need to decide which is the main one to record. Funding: Financial support for this study was provided in part by a grant from the European Commission under the 7 th Framework Programme, Grant Agreement Number for Translational Research and Patient Safety in Europe (TRANSFoRm - Financial support for the study that elicited the think-aloud protocols was provided by CRUK to Olga Kostopoulou, under the NAEDI scheme (ref. C33754/A12222). Financial support for the study that elicited the CDM interview protocols was provided by a departmental PhD studentship to Amanda Woolley. Ethical approval: Ethical approvals were obtained from the North West London Research Ethics Committee 2 (10/H0720/50) and West London Research Ethics Committee 2 (11/LO/0079). Competing interests: The authors have no competing interests to declare. Acknowledgements: Ellen Wright and Thomas Round helped with piloting the follow up interviews, recruiting other GPs for observations and interviewing and providing clinical advice. References 1. Silk N. What went wrong in 1000 negligence claims. Health Care Risk Report, November 2000: Gandhi TK, Kachalia A, Thomas EJ, et al. Missed and Delayed Diagnoses in the Ambulatory Setting: A Study of Closed Malpractice Claims. Ann Intern Med 2006;145(7):

64 3. Graber ML, Kissam S, Payne VL, et al. Cognitive interventions to reduce diagnostic error: a narrative review. Bmj Qual Saf 2012;21(7): de Dombal FT, Leaper DJ, Staniland JR, et al. Computer-aided diagnosis of acute abdominal pain. BMJ 1972;2: Shibl R, Lawley M, Debuse J. Factors influencing decision support system acceptance. Decis Support Syst 2013;54(2): Kawamoto K, Houlihan CA, Balas EA, et al. Improving clinical practice using clinical decision support systems: a systematic review of trials to identify features critical to success. BMJ 2005;330(7494): Garg AX, Adhikari NKJ, McDonald H, et al. Effects of computerized clinical decision support systems on practitioner performance and patient outcomes - A systematic review. JAMA 2005;293(10): Friedman CP, Gatti GG, Franz TM, et al. Do Physicians Know When Their Diagnoses Are Correct? Implications for Decision Support and Error Reduction. J Gen Intern Med 2005;20(4): Friedman CP, Elstein AS, Wolf FM, et al. Enhancement of Clinicians' Diagnostic Reasoning by Computer-Based Consultation: A Multisite Study of 2 Systems. JAMA 1999;282(19): Kostopoulou O, Rosen A, Round T, et al. Early diagnostic suggestions improve the diagnostic accuracy of Family Physicians: a randomized controlled trial (under review). 11. Militello LG, Klein G. Decision centered design. In: Lee J, Kirlik A, eds. Oxford Handbook of Cognitive Engineering, Chipman SF, Schraagen JM, Shalin VL. Introduction to cognitive task analysis. In: Schraagen JM, Chipman SF, Shalin VL, eds. Cognitive task analysis. Mahwah, NJ: Lawrence Erlbaum Associates, 2000: Kostopoulou O. From cognition to the system: developing a multilevel taxonomy of patient safety in general practice. Ergonomics 2006;49(5-6): Shepherd A. Hierarchical Task Analysis. London: Taylor & Francis, Ericsson KA, Moxley JH. Thinking aloud protocols: Concurrent verbalizations of thinking during performance on tasks involving decision making. In: Schulte-Mecklenbeck M, Kühberger A, Ranyard R, eds. A handbook of process tracing methods for decision research: A critical review and user's guide. Hove: Psychology Press, Hoffman RR, Crandall B, Shadbolt N. Use of the Critical Decision Method to Elicit Expert Knowledge: A Case Study in the Methodology of Cognitive Task Analysis. Human Factors 1998;40(2): Woolley A, Kostopoulou O. Clinical intuition in family medicine: more than first impressions. Ann Fam Med 2013;11(1): Klein GA, Calderwood R, Macgregor D. Critical Decision Method for Eliciting Knowledge. Ieee T Syst Man Cyb 1989;19(3): Stanton NA, Chambers PRG, Piggott J. Situational awareness and safety. Saf Sci 2001;39: Elstein AS, Shulman LS, Sprafka SA. Medical Problem Solving: A Ten-Year Retrospective. Evaluation and the Health Professions 1990;13(1): Elstein AS, Shulman LS, Sprafka SA. Medical problem solving: An analysis of clinical reasoning. Cambridge, MA: Harvard University Press, Evans JST, Stanovich KE. Dual-Process Theories of Higher Cognition: Advancing the Debate. Perspect Psychol Sci 2013;8(3):

65 Appendix Cognitive Requirements Table Cognitive requirement Difficulty/potential errors How is it done? Critical cues Design solutions 1. Retrieval and integration of information (current information in the ehr with the RfE) Important information in the ehr may be missed. Early assumptions may bias the interpretation of the RfE. Before patient enters, GPs scan the ehr. GPs try to listen to the RfE without interrupting. Previous consultations, risk factors, chronic diseases, current medications, recent test results 1. Make critical cues in the ehr more salient. Use easily recognisable icons and combined displays. 2. Diagnostic hypotheses generation Focusing narrowly on a hypothesis. GPs may think in terms of general categories (serious/not serious, urgent/not urgent). They may also consciously try to think of and exclude serious but rare conditions. As above Suggest possible diagnoses by integrating information from the ehr with the RfE. 3. Hypothesis testing and refinement Difficulty integrating new information that is not consistent with leading hypothesis. See below. Working hypothesis, patient input, History taking, physical examination, current and past investigations Provide an easy interface for entering coded symptoms and signs. Update the list of suggested diagnoses according to coded input of symptoms and signs What questions to ask the patient? GPs may omit to check for important symptoms and signs of serious but rarer diseases. Asking about alarm symptoms to rule out serious diagnoses. As above. Enable users to click on suggested diagnoses and view important features (that impact on diagnostic likelihood). Propose examinations & investigations that could differentiate between the suggested diagnoses. 65

66 Cognitive requirement Difficulty/potential errors How is it done? Critical cues Design solutions 3.2. If and what investigations to perform or order? Not investigating appropriately (ordering too few or too many investigations). Knowing the diagnostic value of investigations. As above How to interpret the information elicited Abnormal results may be normalised or dismissed. Patient attributes, age, gender, risk factors. Contextualise abnormal or borderline results according to patient s age, RfE and possible diagnoses When to stop eliciting information (define a stopping rule) Too little information may be elicited. Too much information may be elicited in the absence of guiding hypotheses Stop searching, once a satisfactory explanation is reached and the most serious alternatives have been explored 3. Alert when GP enters a diagnosis without having checked symptoms and signs of lifethreatening or urgent conditions 4. Deciding on a course of action Inappropriate management due to misdiagnosis or lack of awareness about managing a specific condition After the user enters a diagnosis, display guidelines with direct links to forms for referral, investigations and prescribing. 66

67 Appendix 2: DDSS User Guide Overview This document describes the functionality of the diagnostic decision support system (DDSS) from your (the GP) point of view. The diagnostic decision support system is a diagnostic tool integrated with the electronic health records (ehr) system to support you in your diagnostic process during consultation. Currently the diagnostic tool is integrated with Vision ehr, but it could integrate with any electronic health records system. This is the first developed prototype, please take into consideration, that it includes the main functionality, but the design and some additions/changes may apply in future versions. Functionality The DDSS enables you to: View a dynamic possible diagnoses list for a specific patient based on the information in the patient s electronic health record and his/her reason for encounter. Select a diagnosis from the list and check what features (symptoms and signs) are relevant, and can change the likelihood of that diagnosis. Code in an easy and intuitive way patient s signs, symptoms and examinations. You can code negative symptoms (e.g., no blood in stool), in addition to positive ones. View recommended examinations and investigations that could be preformed in order to differentiate between the diagnoses on the list. View the reason/s why a specific diagnosis is on the possible diagnoses list. Select a diagnosis. Workflow In the following we will describe the diagnostic process when using the diagnostic tool. 1. View patient information As done today, open the patient s electronic health record in Vision (Figure 1). 67

68 Figure 1 A simulated patient s health record in Vision ehr 2. Evoke the diagnostic tool and select RfE If the patient s reason for encounter (RfE) is diagnostic (not administrative, e.g., requesting a repeated prescription), click the New consultation button from the ehr toolbar (Figure 2), a dialog requesting you to enter the patient s main reason for encounter will be displayed (Figure 3) on top of the patient s health record. You can always navigate from and to the patient s health record. Figure 2: New Consultation button is added to Vision toolbar (first button on the left) Currently the diagnostic tool supports three reasons for encounter (RfEs): abdominal pain, chest pain and dyspnoea. The RfE field in the RfE dialog is an auto-complete field. Once you start typing, the relevant options will be displayed (see Figure 3). Evoking the continue button will close the RfE dialog and open the main diagnostic screen (figure 5). You can minimize or enlarge the screen by clicking the relevant icons on the right, and you can close the screen by clicking the X icon. Figure 3: Reason for encounter (RfE) dialog 68

69 3. Diagnostic support The main diagnostic screen contains the following zones and components: screen caption; significant patient information; patient s RfE; signs, symptoms and examinations list; general notes/comments; possible diagnoses list; and operation button (Figure 4). Figure 4: diagnostic screen application shell Figure 5: main diagnostic screen 3.1 Screen caption The screen caption contains the patient name, gender, and DoB. You can minimize or enlarge the screen by clicking the relevant icons on the right. You can close the 69

70 screen by clicking the X icon. A message asking if you are sure that you want to exit the diagnostic support will be displayed. Figure 6: Screen caption 3.2 Significant patient information On the top left side of the screen, significant information about the patient, which will be taken from the health record such as risk factors (e.g., relevant family history, smoking and alcohol usage, previous cancer), and chronic diseases (e.g., diabetes, asthma), will be displayed. Hovering on a cue will display more information (e.g., hovering on smoking will display a tooltip: 10 packages a day ). Figure 7: Significant patient information area 3.3 Patient s RfE The reason for encounter that you selected in the previous screen (e.g., abdominal pain) is now displayed on top of this screen. You can edit the reason for encounter by clicking Edit. The RfE dialog will open and enable you to change the reason for encounter (Figure 8). Figure 8: The RfE dialog is displayed after clicking Edit. 3.4 Signs, symptoms and examinations list 70

71 In this part of the screen, you can enter coded patient s signs, symptoms and examination results. This field is an auto-complete field, once you start typing, the relevant options will be displayed (see Figure 9). It is important to note that we filtered the list of all possible Read Codes, to a reasonable size, enabling you to select frequent used and relevant signs and symptoms, without the need to search from an enormous database. For example, instead of including the 21 Read Codes that are related to a pregnant woman (e.g., wife pregnant, partner pregnant, pregnant IUD failure), we display only one Read Code: Patient pregnant. Figure 9: Entering patient s signs, symptoms and examination results. You can enter information using the mouse or the keyboard. Using the keyboard, you can start typing the desired symptom, sign or examination (e.g., vomi for vomiting), navigate to the desired value using the arrows (e.g., Diarrhoea and vomiting ), and press the enter key to select the value. Pressing the enter key again will evoke the Add button and display a small dialog enabling you to write notes about the specific sign, symptom or examination (Figure 10). After writing the notes, pressing the enter key again (evoking the OK button), will display the sign, symptom or examination in the list (see Figure 11). 71

72 Figure 10: Evoking the Add button displays the Notes dialog. Figure 11: Evoking the OK button from the notes dialog adds the selected value (e.g., vomiting) to the list. For each sign and symptom you can indicate whether it is present or absent, by ticking the Yes or No checkboxes ( Yes is the default value). For example, you can select the symptom blood in vomit and indicate yes if it exists and No if it doesn t (see Figure 12). 72

73 Figure 12: No blood in vomit symptom was added to the list (the No value is checked). Each time you enter a new value (sign, symptom or examination result), if it is relevant to a specific diagnosis or several diagnoses, the possible diagnoses list on the right side will update accordingly. You can remove a value by clicking the red X icon on the left side of the value. 3.5 General notes In this text box (Figure 13) you can enter general comments or notes concerning the consultation, which are not related to a specific sign or symptom. Figure 13: General notes/comments text field. 3.6 Possible diagnoses list On the right side of the screen, a list displaying the possible diagnoses for a specific patient, ranked according to likelihood, is displayed (Figures 12, 14). The list of possible diagnoses, is based on the information from the patient s electronic health record such as: demographic data (age, gender, etc.), risk factors (smoking, alcohol, previous cancer, BP, etc.), previous diagnoses, chronic health problems (e.g., asthma, diabetics), and recent investigation results (e.g., relevant blood tests) and the patient s reason for encounter you entered in the first step. The list updates automatically as you gather and enter information to the signs, symptoms and 73

74 examinations list. Figure 14: Possible diagnose list (the list is ranked according to likelihood). For each diagnosis on the list you can see the reasons this diagnosis is displayed on the list (for example: female, over 60 years old, etc.), by hovering on the diagnosis. Figure 15: Hovering on a diagnosis will display the reasons this diagnosis is on the list You can select a diagnosis from the list and check what signs and symptoms are relevant and can change the likelihood of that diagnosis (see Figure 16). For example clicking on Crohn s disease, will display: diarrhoea, weight loss, anal fissures, mouth ulcers, etc.. For each symptom you can tick Yes or No, the symptoms you checked will be added to the signs, symptoms and examinations list (see Figure 16). Clicking on the diagnosis again will update the possible diagnoses list according to the new information you entered. 74

75 Figure 16: Clicking on a diagnosis will display the main symptoms that can confirm it or rule it out It is important to note that only reason for encounter is a mandatory field. However, entering patient s signs, symptoms and examinations is recommended in order for the system to be more efficient in its diagnoses support. 4. Decide on a diagnosis Clicking Done from the main diagnostic screen (Figure 16) will display the Working diagnosis dialog (Figure 17). In this screen you can select the working diagnosis for the patient and add notes. The diagnosis field is an auto-complete field (similar to the RfE field), once you start typing, the relevant options will be displayed. You can interact with this dialog using the mouse or the keyboard. Using the keyboard, you can start typing the desired diagnosis (e.g., appen for appendicitis), select the desired value using the arrows, and press the enter key to select the desired value. You can add notes to the diagnosis, such as differentials and/or management plans. 75

76 Figure 17: Working diagnosis dialog 5. Transfer the information Clicking the OK button will close the diagnostic tool and transfer all the information you entered to the patient s health record in vision (see figure 18). Now you can proceed with the consultation: prescribe medications, refer to diagnostic tests, etc. New Added Information Figure 18: patient s health record containing the information transferred for the diagnostic tool During the consultation, you can go back and forth from the diagnostic tool to the 76

77 patient health record in Vision. 6. Premature closure alert The premature alert is displayed only after you give a diagnosis but failed to check at least one of the life-threatening diseases symptoms and signs. The system will display the alert once you close the DDSS (click the Done button). The screen will display only the serious diagnoses (life-threatening ones, see Figure 19 for a preliminary design). You can decide to go back to the main diagnostic screen (click reconsider ) and check the diseases relevant symptoms (see Figure 16) or you may ignore the alert by clicking the Ignore button. Figure 19: Premature closure alert (preliminary design) 77

Original citation: Lim Choi Keung, Sarah Niukyun, Zhao, Lei, Curcin, Vasa, Ethier, Jean-François, Burgun, Anita, McGilchrist, Mark, Bródka, Piotr, Tuligłowicz, Włodzimierz, Andreasson, Anna, Delaney, Brendan

More information

OECD QSAR Toolbox v.4.2. An example illustrating RAAF scenario 6 and related assessment elements

OECD QSAR Toolbox v.4.2. An example illustrating RAAF scenario 6 and related assessment elements OECD QSAR Toolbox v.4.2 An example illustrating RAAF scenario 6 and related assessment elements Outlook Background Objectives Specific Aims Read Across Assessment Framework (RAAF) The exercise Workflow

More information

PROPOSED WORK PROGRAMME FOR THE CLEARING-HOUSE MECHANISM IN SUPPORT OF THE STRATEGIC PLAN FOR BIODIVERSITY Note by the Executive Secretary

PROPOSED WORK PROGRAMME FOR THE CLEARING-HOUSE MECHANISM IN SUPPORT OF THE STRATEGIC PLAN FOR BIODIVERSITY Note by the Executive Secretary CBD Distr. GENERAL UNEP/CBD/COP/11/31 30 July 2012 ORIGINAL: ENGLISH CONFERENCE OF THE PARTIES TO THE CONVENTION ON BIOLOGICAL DIVERSITY Eleventh meeting Hyderabad, India, 8 19 October 2012 Item 3.2 of

More information

Clay Tablet Connector for hybris. User Guide. Version 1.5.0

Clay Tablet Connector for hybris. User Guide. Version 1.5.0 Clay Tablet Connector for hybris User Guide Version 1.5.0 August 4, 2016 Copyright Copyright 2005-2016 Clay Tablet Technologies Inc. All rights reserved. All rights reserved. This document and its content

More information

Combining Archetypes with Fast Health Interoperability Resources in Future-proof Health Information Systems

Combining Archetypes with Fast Health Interoperability Resources in Future-proof Health Information Systems 180 Digital Healthcare Empowering Europeans R. Cornet et al. (Eds.) 2015 European Federation for Medical Informatics (EFMI). This article is published online with Open Access by IOS Press and distributed

More information

The openehr approach. Background. Approach. Heather Leslie, July 17, 2014

The openehr approach. Background. Approach. Heather Leslie, July 17, 2014 Heather Leslie, July 17, 2014 The openehr approach Background The open source openehr 1 architecture specifications 2 have evolved as the result of over 20 years of research and international implementations,

More information

SAGE. Nick Beard Vice President, IDX Systems Corp.

SAGE. Nick Beard Vice President, IDX Systems Corp. SAGE Nick Beard Vice President, IDX Systems Corp. Sharable Active Guideline Environment An R&D consortium to develop the technology infrastructure to enable computable clinical guidelines, that will be

More information

Lionbridge Connector for Hybris. User Guide

Lionbridge Connector for Hybris. User Guide Lionbridge Connector for Hybris User Guide Version 2.1.0 November 24, 2017 Copyright Copyright 2017 Lionbridge Technologies, Inc. All rights reserved. Published in the USA. March, 2016. Lionbridge and

More information

Clinical Observation Modeling

Clinical Observation Modeling Clinical Observation Modeling VA Informatics Architecture SOLOR Meeting Walter Sujansky January 31, 2013 Goals of Clinical Observation Modeling Create conceptual-level models of the discrete statements

More information

The OUTBACK Trial. Specific CRF Completion Guidelines

The OUTBACK Trial. Specific CRF Completion Guidelines The OUTBACK Trial A Phase III trial of adjuvant chemotherapy following chemoradiation as primary treatment for locally advanced cervical cancer compared to chemoradiation alone (ANZGOG 0902, GOG 0274,

More information

Incorporation of Imaging-Based Functional Assessment Procedures into the DICOM Standard Draft version 0.1 7/27/2011

Incorporation of Imaging-Based Functional Assessment Procedures into the DICOM Standard Draft version 0.1 7/27/2011 Incorporation of Imaging-Based Functional Assessment Procedures into the DICOM Standard Draft version 0.1 7/27/2011 I. Purpose Drawing from the profile development of the QIBA-fMRI Technical Committee,

More information

Investigating implementing CEN with HL7 V3 and SNOMED CT Final Report

Investigating implementing CEN with HL7 V3 and SNOMED CT Final Report Investigating implementing CEN 13606 with HL7 V3 and SNOMED CT Programme NPFIT Document Record ID Key Sub-Prog / Project Technology Office Prog. Director P. Jones Status

More information

BlueBayCT - Warfarin User Guide

BlueBayCT - Warfarin User Guide BlueBayCT - Warfarin User Guide December 2012 Help Desk 0845 5211241 Contents Getting Started... 1 Before you start... 1 About this guide... 1 Conventions... 1 Notes... 1 Warfarin Management... 2 New INR/Warfarin

More information

The future of ODM. FH-Prof. Dr. Jozef Aerts University of Applied Sciences FH Joanneum Graz, Austria

The future of ODM. FH-Prof. Dr. Jozef Aerts University of Applied Sciences FH Joanneum Graz, Austria The future of ODM FH-Prof. Dr. Jozef Aerts University of Applied Sciences FH Joanneum Graz, Austria History of the ODM Standard ODM = Operational Data Model XML-based standard for design, exchange, and

More information

Anamnesis via the Internet - Prospects and Pilot Results

Anamnesis via the Internet - Prospects and Pilot Results MEDINFO 2001 V. Patel et al. (Eds) Amsterdam: IOS Press 2001 IMIA. All rights reserved Anamnesis via the Internet - Prospects and Pilot Results Athanasios Emmanouil and Gunnar O. Klein Centre for Health

More information

Polypharmacy and Deprescribing. A special report on views from the PrescQIPP landscape review

Polypharmacy and Deprescribing. A special report on views from the PrescQIPP landscape review Polypharmacy and Deprescribing A special report on views from the PrescQIPP landscape review Introduction and background In recent years, we have seen the emergence of an international discussion around

More information

Cerner COMPASS ICD-10 Transition Guide

Cerner COMPASS ICD-10 Transition Guide Cerner COMPASS ICD-10 Transition Guide Dx Assistant Purpose: To educate Seton clinicians regarding workflow changes within Cerner COMPASS subsequent to ICD-10 transition. Scope: Basic modules and functionality

More information

MyDispense OTC exercise Guide

MyDispense OTC exercise Guide MyDispense OTC exercise Guide Version 5.0 Page 1 of 23 Page 2 of 23 Table of Contents What is MyDispense?... 4 Who is this guide for?... 4 How should I use this guide?... 4 OTC exercises explained... 4

More information

The Hospital Anxiety and Depression Scale Guidance and Information

The Hospital Anxiety and Depression Scale Guidance and Information The Hospital Anxiety and Depression Scale Guidance and Information About Testwise Testwise is the powerful online testing platform developed by GL Assessment to host its digital tests. Many of GL Assessment

More information

Collaborative Project of the 7th Framework Programme. WP6: Tools for bio-researchers and clinicians

Collaborative Project of the 7th Framework Programme. WP6: Tools for bio-researchers and clinicians G.A. nº 270086 Collaborative Project of the 7th Framework Programme WP6: Tools for bio-researchers and clinicians Deliverable 6.1: Design of Inference Engine v.1.0 31/10/2011 www.synergy-copd.eu Document

More information

NIH NEW CLINICAL TRIAL REQUIREMENTS AND FORMS-E

NIH NEW CLINICAL TRIAL REQUIREMENTS AND FORMS-E NIH NEW CLINICAL TRIAL REQUIREMENTS AND FORMS-E HOW DOES THE NIH DEFINE CLINICAL TRIAL AND HOW DO I KNOW IF MY STUDY IS ONE? 01-18-18 A clinical trial is defined by the NIH as a research study in which

More information

Content Part 2 Users manual... 4

Content Part 2 Users manual... 4 Content Part 2 Users manual... 4 Introduction. What is Kleos... 4 Case management... 5 Identity management... 9 Document management... 11 Document generation... 15 e-mail management... 15 Installation

More information

Entering HIV Testing Data into EvaluationWeb

Entering HIV Testing Data into EvaluationWeb Entering HIV Testing Data into EvaluationWeb User Guide Luther Consulting, LLC July, 2014/v2.2 All rights reserved. Table of Contents Introduction... 3 Accessing the CTR Form... 4 Overview of the CTR Form...

More information

General Practice Extraction Service (GPES)

General Practice Extraction Service (GPES) 7 General Practice Extraction Service (GPES) Customer Requirement Summary (for aggregate data extractions) Customer: NHS England Requirement: Pertussis Enhanced Service for financial year 2015-16 Customer

More information

Appendix B. Nodulus Observer XT Instructional Guide. 1. Setting up your project p. 2. a. Observation p. 2. b. Subjects, behaviors and coding p.

Appendix B. Nodulus Observer XT Instructional Guide. 1. Setting up your project p. 2. a. Observation p. 2. b. Subjects, behaviors and coding p. 1 Appendix B Nodulus Observer XT Instructional Guide Sections: 1. Setting up your project p. 2 a. Observation p. 2 b. Subjects, behaviors and coding p. 3 c. Independent variables p. 4 2. Carry out an observation

More information

Identifying patients who may benefit from stepping down PPI treatment

Identifying patients who may benefit from stepping down PPI treatment CLINICAL AUDIT Identifying patients who may benefit from stepping down PPI treatment Valid to January 2024 bpac nz better medicin e This audit identifies patients who are prescribed the proton pump inhibitor

More information

An Ontology for Healthcare Quality Indicators: Challenges for Semantic Interoperability

An Ontology for Healthcare Quality Indicators: Challenges for Semantic Interoperability 414 Digital Healthcare Empowering Europeans R. Cornet et al. (Eds.) 2015 European Federation for Medical Informatics (EFMI). This article is published online with Open Access by IOS Press and distributed

More information

Dementia Direct Enhanced Service

Dementia Direct Enhanced Service Vision 3 Dementia Direct Enhanced Service England Outcomes Manager Copyright INPS Ltd 2015 The Bread Factory, 1A Broughton Street, Battersea, London, SW8 3QJ T: +44 (0) 207 501700 F:+44 (0) 207 5017100

More information

Development of an Expert System for Distinguishing Headaches from Migraines

Development of an Expert System for Distinguishing Headaches from Migraines Development of an Expert System for Distinguishing Headaches from Migraines By D. KOPEC*, G. SHAGAS*, J. SELMAN**, D. REINHARTH**, S. TAMANG* * Department of Computer and Information Science, Brooklyn

More information

Getting the Design Right Daniel Luna, Mackenzie Miller, Saloni Parikh, Ben Tebbs

Getting the Design Right Daniel Luna, Mackenzie Miller, Saloni Parikh, Ben Tebbs Meet the Team Getting the Design Right Daniel Luna, Mackenzie Miller, Saloni Parikh, Ben Tebbs Mackenzie Miller: Project Manager Daniel Luna: Research Coordinator Saloni Parikh: User Interface Designer

More information

Corporate Online. Using Term Deposits

Corporate Online. Using Term Deposits Corporate Online. Using Term Deposits About this Guide About Corporate Online Westpac Corporate Online is an internet-based electronic platform, providing a single point of entry to a suite of online transactional

More information

Managing Immunizations

Managing Immunizations Managing Immunizations In this chapter: Viewing Immunization Information Entering Immunizations Editing Immunizations Entering a Lead Test Action Editing a Lead Test Action Entering Opt-Out Immunizations

More information

2016 Children and young people s inpatient and day case survey

2016 Children and young people s inpatient and day case survey NHS Patient Survey Programme 2016 Children and young people s inpatient and day case survey Technical details for analysing trust-level results Published November 2017 CQC publication Contents 1. Introduction...

More information

Creating a reminder for GSK vaccines in Allscripts Professional EHR

Creating a reminder for GSK vaccines in Allscripts Professional EHR Vaccine reminders Ensure timely and appropriate vaccine orders and administration Keeping track of patient vaccination requirements is an important aspect in the delivery of ongoing patient care. Alerts

More information

Enhanced Asthma Management with Mobile Communication

Enhanced Asthma Management with Mobile Communication Enhanced Asthma Management with Mobile Communication P.S. Ngai, S. Chan, C.T. Lau, K.M. Lau Abstract In this paper, we propose a prototype system to enhance the management of asthma condition in patients

More information

BORN Ontario: NSO DERF Training Guide FEBRUARY 2012

BORN Ontario: NSO DERF Training Guide FEBRUARY 2012 BORN Ontario: NSO DERF Training Guide FEBRUARY 2012 NSO DERF Encounter NSO sends newborn screening records to the BORN Information System (BIS) regularly throughout each day. When a risk letter is printed,

More information

IMPaLA tutorial.

IMPaLA tutorial. IMPaLA tutorial http://impala.molgen.mpg.de/ 1. Introduction IMPaLA is a web tool, developed for integrated pathway analysis of metabolomics data alongside gene expression or protein abundance data. It

More information

ORC: an Ontology Reasoning Component for Diabetes

ORC: an Ontology Reasoning Component for Diabetes ORC: an Ontology Reasoning Component for Diabetes Özgür Kafalı 1, Michal Sindlar 2, Tom van der Weide 2 and Kostas Stathis 1 1 Department of Computer Science Royal Holloway, University of London, UK {ozgur.kafali,kostas.stathis}@rhul.ac.uk

More information

Standard operating procedure

Standard operating procedure Standard operating procedure Title: Preparation of an initial European Public Assessment Report (EPAR) for a human medicinal product following positive or negative opinion Status: PUBLIC Document no.:

More information

Quality requirements for EHR Archetypes

Quality requirements for EHR Archetypes Quality requirements for EHR Archetypes Dr Archana Tapuria, UCL Dr Tony Austin, UCL Prof Dipak Kalra, UCL, EuroRec Prof Georges De Moor, Univ. Gent, EuroRec MIE 2012, Pisa What is an Electronic Health

More information

HL7 s Version 2 standards and submission to immunization registries

HL7 s Version 2 standards and submission to immunization registries HL7 s Version 2 standards and submission to immunization registries Alean Kirnak Cochair HL7 Public Health and Emergency Response Workgroup American Immunization Registry Association Software Partners

More information

Institute of Medicine Workshop: DIGITAL DATA PRIORITIES FOR CONTINUOUS LEARNING IN HEALTH AND HEALTH CARE

Institute of Medicine Workshop: DIGITAL DATA PRIORITIES FOR CONTINUOUS LEARNING IN HEALTH AND HEALTH CARE Institute of Medicine Workshop: DIGITAL DATA PRIORITIES FOR CONTINUOUS LEARNING IN HEALTH AND HEALTH CARE Data Quality in Clinical Research 23 March 2012 Rebecca D. Kush, PhD President and CEO, CDISC CDISC

More information

Consultation on publication of new cancer waiting times statistics Summary Feedback Report

Consultation on publication of new cancer waiting times statistics Summary Feedback Report Consultation on publication of new cancer waiting times statistics Summary Feedback Report Information Services Division (ISD) NHS National Services Scotland March 2010 An electronic version of this document

More information

Using the NIH Collaboratory's and PCORnet's distributed data networks for clinical trials and observational research - A preview

Using the NIH Collaboratory's and PCORnet's distributed data networks for clinical trials and observational research - A preview Using the NIH Collaboratory's and PCORnet's distributed data networks for clinical trials and observational research - A preview Millions of people. Strong collaborations. Privacy first. Jeffrey Brown,

More information

EPF s response to the European Commission s public consultation on the "Summary of Clinical Trial Results for Laypersons"

EPF s response to the European Commission s public consultation on the Summary of Clinical Trial Results for Laypersons EPF s response to the European Commission s public consultation on the "Summary of Clinical Trial Results for Laypersons" August 2016 This document received funding under an operating grant from the European

More information

Evaluation: Controlled Experiments. Title Text

Evaluation: Controlled Experiments. Title Text Evaluation: Controlled Experiments Title Text 1 Outline Evaluation beyond usability tests Controlled Experiments Other Evaluation Methods 2 Evaluation Beyond Usability Tests 3 Usability Evaluation (last

More information

USER GUIDE: NEW CIR APP. Technician User Guide

USER GUIDE: NEW CIR APP. Technician User Guide USER GUIDE: NEW CIR APP. Technician User Guide 0 Table of Contents 1 A New CIR User Interface Why?... 3 2 How to get started?... 3 3 Navigating the new CIR app. user interface... 6 3.1 Introduction...

More information

Incorporation of Imaging-Based Functional Assessment Procedures into the DICOM Standard Draft version 0.2 8/10/2011 I. Purpose

Incorporation of Imaging-Based Functional Assessment Procedures into the DICOM Standard Draft version 0.2 8/10/2011 I. Purpose Incorporation of Imaging-Based Functional Assessment Procedures into the DICOM Standard Draft version 0.2 8/10/2011 I. Purpose Drawing from the profile development of the QIBA-fMRI Technical Committee,

More information

Automated Immunization Evaluation Process

Automated Immunization Evaluation Process University of Medicine and Dentistry of New Jersey - University of Pennsylvania New Jersey Comprehensive Immunization Program (NJ-CIP) Automated Immunization Evaluation Process Working Paper rev. May 11,

More information

Known Issue Summary Topic Found in Version

Known Issue Summary Topic Found in Version Advanced Audit Audit KI90525 Messaging Service Log files in the C:\NextGen\Services\.NGPr od.auditmessaging\logs are generate logs extremely fast. When opening the log files it is filled

More information

About this consent form

About this consent form Protocol Title: Development of the smoking cessation app Smiling instead of Smoking Principal Investigator: Bettina B. Hoeppner, Ph.D. Site Principal Investigator: n/a Description of Subject Population:

More information

Expert System Profile

Expert System Profile Expert System Profile GENERAL Domain: Medical Main General Function: Diagnosis System Name: INTERNIST-I/ CADUCEUS (or INTERNIST-II) Dates: 1970 s 1980 s Researchers: Ph.D. Harry Pople, M.D. Jack D. Myers

More information

PROJECT PERIODIC REPORT

PROJECT PERIODIC REPORT PROJECT PERIODIC REPORT Project acronym: Project full title: Grant agreement no: CuPiD Closed-loop system for personalized and at-home rehabilitation of people with Parkinson's Disease ICT 288516 Project

More information

Using the NWD Integrative Screener as a Data Collection Tool Agency-Level Aggregate Workbook

Using the NWD Integrative Screener as a Data Collection Tool Agency-Level Aggregate Workbook Using the NWD Integrative Screener as a Data Collection Tool Agency-Level Aggregate Workbook The NWD Integrative Screener was designed to be user friendly for providers and agencies. Many items and health

More information

Alcohol interventions in secondary and further education

Alcohol interventions in secondary and further education National Institute for Health and Care Excellence Guideline version (Draft for Consultation) Alcohol interventions in secondary and further education NICE guideline: methods NICE guideline Methods

More information

Warfarin Help Documentation

Warfarin Help Documentation Warfarin Help Documentation Table Of Contents Warfarin Management... 1 iii Warfarin Management Warfarin Management The Warfarin Management module is a powerful tool for monitoring INR results and advising

More information

CLINICIAN-LED E-HEALTH RECORDS. (AKA GETTING THE LITTLE DATA RIGHT) Dr Heather Leslie Ocean Informatics/openEHR Foundation

CLINICIAN-LED E-HEALTH RECORDS. (AKA GETTING THE LITTLE DATA RIGHT) Dr Heather Leslie Ocean Informatics/openEHR Foundation CLINICIAN-LED E-HEALTH RECORDS (AKA GETTING THE LITTLE DATA RIGHT) Dr Heather Leslie Ocean Informatics/openEHR Foundation An ongoing issue In attempting to arrive at the truth, I have applied everywhere

More information

QUALITY IMPROVEMENT TOOLS

QUALITY IMPROVEMENT TOOLS QUALITY IMPROVEMENT TOOLS QUALITY IMPROVEMENT TOOLS The goal of this section is to build the capacity of quality improvement staff to implement proven strategies and techniques within their health care

More information

Project Nexus Market Trials Regression Test Approach

Project Nexus Market Trials Regression Test Approach 2016 Project Nexus Market Trials Regression Test Approach V1.0 FINAL DRAFT Table of Contents 1 - INTRODUCTION... 2 2 - CONTEXT... 2 3 - MT REGRESSION OBJECTIVES AND SCOPE... 2 4 - DURATION AND PLAN...

More information

Language Access Plan Basics

Language Access Plan Basics Language Access Plan Basics What you need to know to build a compliant and effective program InDemand Interpreting 01 Compliance HOW TO COMPLY WITH UPDATED LANGUAGE ACCESS STANDARDS Federal mandates require

More information

AUDIT OUTLINE INFORMATION SUMMARY

AUDIT OUTLINE INFORMATION SUMMARY AUDIT OUTLINE INFORMATION SUMMARY 1. External QA Each DAFNE centre will undergo an external audit visit every 3 years. The external audit visit will take place during a week that the centre being audited

More information

CHRONIC CARE REPORTS. V 9.0, October eclinicalworks, All rights reserved

CHRONIC CARE REPORTS. V 9.0, October eclinicalworks, All rights reserved CHRONIC CARE REPORTS V 9.0, October 2011 eclinicalworks, 2011. All rights reserved CONTENTS ABOUT THIS GUIDE 3 Product Documentation 3 Finding the Documents 3 Webinars 3 eclinicalworks Newsletter 4 Getting

More information

How Doctors Feel About Electronic Health Records. National Physician Poll by The Harris Poll

How Doctors Feel About Electronic Health Records. National Physician Poll by The Harris Poll How Doctors Feel About Electronic Health Records National Physician Poll by The Harris Poll 1 Background, Objectives, and Methodology New research from Stanford Medicine, conducted with The Harris Poll

More information

Automated process to create snapshot reports based on the 2016 Murray Community-based Groups Capacity Survey: User Guide Report No.

Automated process to create snapshot reports based on the 2016 Murray Community-based Groups Capacity Survey: User Guide Report No. research for a sustainable future Automated process to create snapshot reports based on the 2016 Murray Community-based Groups Capacity Survey: User Guide Report No. 116 Steven Vella Gail Fuller Michael

More information

UWA ERA Publications Collection 2011

UWA ERA Publications Collection 2011 UWA ERA Publications Collection 2011 Overview of the collection process Introduction to Minerva Research Assessment Unit September 2011 In this seminar Why we collect ERA publications data See UWA Publications

More information

The MetroHealth System. Creating the HIT Organizational Culture at MetroHealth. Creating the HIT Organizational Culture

The MetroHealth System. Creating the HIT Organizational Culture at MetroHealth. Creating the HIT Organizational Culture CASE STUDY CASE STUDY The MetroHealth System Optimizing Health Information Technology to Increase Vaccination Rates The MetroHealth System in Cleveland, Ohio, was the first safety-net health care system

More information

Updating immunization schedules to reflect GSK vaccines

Updating immunization schedules to reflect GSK vaccines in Professional EHR Manually updating immunization schedules in Professional Updating immunization schedules to reflect GSK vaccines E-Prescribing vaccines in Epic EHR The Centers for Disease Control and

More information

CHAPTER 6 DESIGN AND ARCHITECTURE OF REAL TIME WEB-CENTRIC TELEHEALTH DIABETES DIAGNOSIS EXPERT SYSTEM

CHAPTER 6 DESIGN AND ARCHITECTURE OF REAL TIME WEB-CENTRIC TELEHEALTH DIABETES DIAGNOSIS EXPERT SYSTEM 87 CHAPTER 6 DESIGN AND ARCHITECTURE OF REAL TIME WEB-CENTRIC TELEHEALTH DIABETES DIAGNOSIS EXPERT SYSTEM 6.1 INTRODUCTION This chapter presents the design and architecture of real time Web centric telehealth

More information

11. NATIONAL DAFNE CLINICAL AND RESEARCH DATABASE

11. NATIONAL DAFNE CLINICAL AND RESEARCH DATABASE 11. NATIONAL DAFNE CLINICAL AND RESEARCH DATABASE The National DAFNE Clinical and Research database was set up as part of the DAFNE QA programme (refer to section 12) to facilitate Audit and was updated

More information

ITIL v3 Service Management as a Practice

ITIL v3 Service Management as a Practice ITIL v3 as a Practice 1 as a Practice ITIL = IT Infrastructure Library Set of books giving guidance on the provision of quality IT services Common language Best practices in delivery of IT services Not

More information

Why Human-Centered Design Matters

Why Human-Centered Design Matters Reading Review: Article Why Human-Centered Design Matters -Dave Thomsen, Wanderful Media HUMAN CENTERED DESIGN a human-centered approach fuels the creation of products that resonate more deeply with an

More information

Primary Health Networks

Primary Health Networks Primary Health Networks Drug and Alcohol Treatment Activity Work Plan 2016-17 to 2018-19 Hunter New England & Central Coast Please note: This Activity Work Plan was developed in response to the HNECC PHN

More information

Web Feature Services Tutorial

Web Feature Services Tutorial Southeast Alaska GIS Library Web Feature Services Tutorial Prepared By Mike Plivelich Version 0.2 Status Draft Updates Continual Release Date June 2010 1 TABLE OF CONTENTS Page # INTRODUCTION...3 PURPOSE:...

More information

A Framework for Conceptualizing, Representing, and Analyzing Distributed Interaction. Dan Suthers

A Framework for Conceptualizing, Representing, and Analyzing Distributed Interaction. Dan Suthers 1 A Framework for Conceptualizing, Representing, and Analyzing Distributed Interaction Dan Suthers Work undertaken with Nathan Dwyer, Richard Medina and Ravi Vatrapu Funded in part by the U.S. National

More information

Hypertension encoded in GLIF

Hypertension encoded in GLIF Hypertension encoded in GLIF Guideline 2 (Based on the hypertension guideline. Simplified (not all contraindications, relative contra-indications, and relative indications are specified). Drug interactions

More information

The MASCC Guidelines Policy

The MASCC Guidelines Policy The MASCC Guidelines Policy Recommendations for MASCC Guideline Construction and the Endorsement of Externally Generated Guidelines Preamble MASCC recognizes that providing supportive care facilitates

More information

The University of Texas MD Anderson Cancer Center Division of Quantitative Sciences Department of Biostatistics. CRM Suite. User s Guide Version 1.0.

The University of Texas MD Anderson Cancer Center Division of Quantitative Sciences Department of Biostatistics. CRM Suite. User s Guide Version 1.0. The University of Texas MD Anderson Cancer Center Division of Quantitative Sciences Department of Biostatistics CRM Suite User s Guide Version 1.0.0 Clift Norris, John Venier, Ying Yuan, and Lin Zhang

More information

Usability Test Report. Tecurologic LLC PediNotes Version 5.1

Usability Test Report. Tecurologic LLC PediNotes Version 5.1 Usability Test Report Tecurologic LLC PediNotes Version 5.1 2 EHR Usability Test Report of PediNotes Version 5.1 Report based on ISO/IEC 25062:2006 Common Industry Format for Usability Test Reports PediNotes

More information

Contribution of Clinical Archetypes, and the Challenges, towards Achieving Semantic Interoperability for EHRs

Contribution of Clinical Archetypes, and the Challenges, towards Achieving Semantic Interoperability for EHRs Case Report Healthc Inform Res. 2013 December;19(4):286-292. pissn 2093-3681 eissn 2093-369X Contribution of Clinical Archetypes, and the Challenges, towards Achieving Semantic Interoperability for EHRs

More information

Standards for Analyzable Submission Datasets for Directed Safety Analyses

Standards for Analyzable Submission Datasets for Directed Safety Analyses Standards for Analyzable Submission Datasets for Directed Safety Analyses Michael Nessly Director, Clinical Biostatistics Merck Research Labs CDISC Analysis Dataset Models Team (ADaM) nesslym@merck.com

More information

Medtech Training Guide

Medtech Training Guide Medtech Training Guide Clinical Audit Tool Copyright Medtech Limited Page 1 of 80 Clinical Audit Tool v4.0 Contents Chapter 1: Getting Started... 3 Logging In... 3 Logging Out... 3 Setting the Preliminary

More information

User Guide for Classification of Diabetes: A search tool for identifying miscoded, misclassified or misdiagnosed patients

User Guide for Classification of Diabetes: A search tool for identifying miscoded, misclassified or misdiagnosed patients User Guide for Classification of Diabetes: A search tool for identifying miscoded, misclassified or misdiagnosed patients For use with isoft Premiere Synergy Produced by André Ring 1 Table of Contents

More information

Clinical Decision Support Technologies for Oncologic Imaging

Clinical Decision Support Technologies for Oncologic Imaging Clinical Decision Support Technologies for Oncologic Imaging Ramin Khorasani, MD, MPH Professor of Radiology Harvard Medical School Distinguished Chair, Medical Informatics Vice Chair, Department of Radiology

More information

Brief Psychiatric Rating Scale-Anchored (BPRS-A)

Brief Psychiatric Rating Scale-Anchored (BPRS-A) Brief Psychiatric Rating Scale-Anchored (BPRS-A) Questionnaire Supplement to the Study Data Tabulation Model Implementation Guide for Human Clinical Trials Prepared by CDISC Questionnaire Sub-team Notes

More information

Charts Worksheet using Excel Obesity Can a New Drug Help?

Charts Worksheet using Excel Obesity Can a New Drug Help? Worksheet using Excel 2000 Obesity Can a New Drug Help? Introduction Obesity is known to be a major health risk. The data here arise from a study which aimed to investigate whether or not a new drug, used

More information

Instructor Guide to EHR Go

Instructor Guide to EHR Go Instructor Guide to EHR Go Introduction... 1 Quick Facts... 1 Creating your Account... 1 Logging in to EHR Go... 5 Adding Faculty Users to EHR Go... 6 Adding Student Users to EHR Go... 8 Library... 9 Patients

More information

BOOKING RULES REVISIONS & SUPPORT FOR MULTIPLE CRS

BOOKING RULES REVISIONS & SUPPORT FOR MULTIPLE CRS CloudPM v5.4 BOOKING RULES REVISIONS & SUPPORT FOR MULTIPLE CRS INTERFACES QUICK REFERENCE GUIDE FOR MSI CLOUDPM USERS ED TRUXAL MSI SOLUTIONS 7600 N. 15th Street Suite #250 Phoenix, AZ 85020 Table of

More information

Introduction. Preliminary POV. Additional Needfinding Results. CS Behavioral Change Studio ASSIGNMENT 2 POVS and Experience Prototypes

Introduction. Preliminary POV. Additional Needfinding Results. CS Behavioral Change Studio ASSIGNMENT 2 POVS and Experience Prototypes CS 147 - Behavioral Change Studio ASSIGNMENT 2 POVS and Experience Prototypes Introduction Meet The BetterMeet Team We are a team of Stanford students in the behavioral change studio. Problem Domain Our

More information

Downloaded from:

Downloaded from: Arnup, SJ; Forbes, AB; Kahan, BC; Morgan, KE; McKenzie, JE (2016) The quality of reporting in cluster randomised crossover trials: proposal for reporting items and an assessment of reporting quality. Trials,

More information

National Cancer Peer Review Programme. Radiotherapy Service Evidence Guide

National Cancer Peer Review Programme. Radiotherapy Service Evidence Guide National Cancer Peer Review Programme Radiotherapy Service Evidence Guide Forward This evidence guide has been formulated to assist organisations in preparing for peer review. The contents of this guide

More information

Using Health IT to Support Oral Health Integration: Dealing with Common Barriers. Jeff Hummel, MD, MPH Qualis Health November 5, 2015

Using Health IT to Support Oral Health Integration: Dealing with Common Barriers. Jeff Hummel, MD, MPH Qualis Health November 5, 2015 Using Health IT to Support Oral Health Integration: Dealing with Common Barriers Jeff Hummel, MD, MPH Qualis Health November 5, 2015 Goals for this Session Understand 3 aspects of oral health information

More information

Anticoagulation Manager - Getting Started

Anticoagulation Manager - Getting Started Vision 3 Anticoagulation Manager - Getting Started Copyright INPS Ltd 2014 The Bread Factory, 1A Broughton Street, Battersea, London, SW8 3QJ T: +44 (0) 207 501700 F:+44 (0) 207 5017100 W: www.inps.co.uk

More information

EHR Usability Test Report of OneTouch EMR

EHR Usability Test Report of OneTouch EMR EHR Usability Test Report of OneTouch EMR Report based on ISO/IEC 25062:2006 Common Industry Format for Usability Test Reports OneTouch EMR, Version 2.0 Date of Usability Test: 5/11/2014 Date of Report:

More information

Survey questionnaire on STI. surveillance, care and prevention. in European countries SAMPLE APPENDIX

Survey questionnaire on STI. surveillance, care and prevention. in European countries SAMPLE APPENDIX European Surveillance of Sexually Transmitted Infections Survey questionnaire on STI surveillance, care and prevention in European countries APPENDIX Detailed questionnaire on clinician and laboratory

More information

OneTouch Reveal Web Application. User Manual for Healthcare Professionals Instructions for Use

OneTouch Reveal Web Application. User Manual for Healthcare Professionals Instructions for Use OneTouch Reveal Web Application User Manual for Healthcare Professionals Instructions for Use Contents 2 Contents Chapter 1: Introduction...4 Product Overview...4 Intended Use...4 System Requirements...

More information

Hospital Anxiety and Depression Scale (HADS)

Hospital Anxiety and Depression Scale (HADS) Hospital Anxiety and Depression Scale (HADS) Questionnaire Supplement to the Study Data Tabulation Model Implementation Guide for Human Clinical Trials Prepared by Business & Decision Life Sciences and

More information

Publishing WFS Services Tutorial

Publishing WFS Services Tutorial Publishing WFS Services Tutorial Copyright 1995-2010 Esri All rights reserved. Table of Contents Tutorial: Publishing a WFS service........................... 3 Copyright 1995-2010 ESRI, Inc. All rights

More information

Agile Product Lifecycle Management for Process

Agile Product Lifecycle Management for Process Nutrition Surveillance Management User Guide Release 5.2.1 Part No. E13901-01 September 2008 Copyrights and Trademarks Copyright 1995, 2008, Oracle Corporation and/or its affiliates. All rights reserved.

More information

Common Criteria. for. CGIAR Research Proposal (CRP) Design and Assessment

Common Criteria. for. CGIAR Research Proposal (CRP) Design and Assessment Common Criteria for CGIAR Research Proposal (CRP) Design and Assessment 30 August 2010 1 CGIAR Research Proposals (CRPs) are reviewed at various stages of their development and approval. These assessments

More information

Semantic Alignment between ICD-11 and SNOMED-CT. By Marcie Wright RHIA, CHDA, CCS

Semantic Alignment between ICD-11 and SNOMED-CT. By Marcie Wright RHIA, CHDA, CCS Semantic Alignment between ICD-11 and SNOMED-CT By Marcie Wright RHIA, CHDA, CCS World Health Organization (WHO) owns and publishes the International Classification of Diseases (ICD) WHO was entrusted

More information