Appendix A: DARTNet Design Specifications

Transcription

1 Appendix A: DARTNet Design Specifications Note: Prepared by the DARTNet Team, University of Colorado Denver Department of Family Medicine, Contract No. HHSA I TO2. A-1

2 Appendix A Contents Introduction... A-3 Overall Aims... A-4 Technical Overview... A-4 Clinical Data Sources... A-5 Clinical Decision Support and Data Extraction... A-7 Data Mapping With Each EHR... A-9 Grid Based Computing... A-9 Data Export and Presentation to the Grid... A-11 Quality Improvement... A-15 Data Additions Through Natural Language Processing... A-15 Directed Point-of-Care Data Collection... A-19 Summary... A-19 References... A-21 Appendix A-A: Criteria for Adding New Member Organizations to DARTNet... A-22 Appendix A-B: Coding Dictionaries for DARTNet Data... A-23 Appendix A-C: About the Globus Toolkit... A-33 Grid Technologies... A-33 Globus Toolkit... A-35 X.509 Security... A-37 Open Grid Services Architecture Data Access and Integration (OGSA-DAI)... A-37 A-2

3 Introduction The Distributed Ambulatory Research in Therapeutics Network (DARTNet) is a prototype federated network of electronic health record (EHR) data from eight organizations representing over 500 clinicians and over 400,000 patients. The prototype system captures, codify and standardize over 150 unique data elements per patient for more than 48 months. DARTNet takes advantage of our team s expertise in analyzing large existing data sets and operating practice-based research networks (PBRN), and is proving to be an asset for the development of a new distributed research network of standardized clinical data from primary care clinicians EHRs. Four current CO-DEcIDE partners were involved in the development of the first prototype for DARTNet: the University of Colorado Department of Family Medicine (CU-DFM), the University of Colorado School of Pharmacy (CU SOP), the American Academy of Family Physicians National Research Network (AAFP NRN) and the Robert Graham Center (RGC). Two technical partners joined in this effort: the University of Minnesota Center for Excellence in Primary Care (UMN) and Clinical Integration Networks of America, Inc. (CINA). The process of developing the DARTNet prototype actively explored how we can use existing EHR data to supplement data from large administrative datasets in order to answer questions concerning the safety and effectiveness of medications and medical devices. Furthermore, using our PBRN expertise assisted us to explore the ability to fill gaps in clinical data using point-of-care data collection techniques. A key requirement for DARTNet is the standardization of data elements across EHR products. We accomplished this using advanced clinical decision support tools already available from CINA. We used tools developed by CINA to access and export standardized data at each clinical organization into a relational data set that we refer to as a Clinical Data Repository (CDR). This standardized data set, which includes patient identifiers, was successfully transferred to a second database (the electronic Primary Care Network Gateway database Gateway for short), de-identified and presented for query access through a secure Grid enabled web-portal. Both of these databases reside within each participating clinical organization. The movement of data from the CDR to the epcrn Gateway database is based on the ASTMstandardized Continuity of Care Record (CCR). A full set of patient data never left the clinical sites where they are stored in this effort; however, the DARTNet team has the ability to query the de-identified federated databases in order to answer research questions that cannot be answered from existing administrative datasets. Furthermore, we explored the development of natural language processing (NLP) system to be used to unlock key data elements from EHR text. The DARTNet system is readily expandable using Grid-based local parallel processing and a two-stage data extraction and de-identification process. The DARTNet architecture will support a final system to accommodate at least two orders per year of magnitude greater than this prototype with a single central technical support site. By adding additional central support sites (or supernodes) the network is essentially infinitely expandable. Furthermore, the data interfaces are not specific to primary care and can be expanded to include sub-specialty data where they are available electronically. When taken to scale, DARTNet will be able to explore both rare safety events in low usage medications and the safety and efficacy of commonly used ambulatory therapies. A-3

4 Overall Aims Aim 1: Develop a federated network of 200+ primary care clinicians who use EHRs, while examining the following issues. a. Establish a governance system that supports access to federated data while allowing members to maintain control of their data. b. Create a data extraction approach that will allow virtually any clinicians with EHRs to join the network as desired. c. Examine the ability of an existing National Institutes of Health (NIH) supported software package to meet the distributed query needs of the network. Aim 2: Analytically demonstrate how existing large-scale data sets can be enhanced by patientlevel data from the federated primary care network to inform and expand knowledge of effective and safe medical therapeutics. a. Use existing large datasets (e.g., Ingenix) to evaluate medical therapeutics safety and effectiveness from a population based level. b. Examine what additional information can be obtained from existing patient level data available through DARTNet. c. Determine what information will only be available through direct data collection from clinicians or patients. Aim 3: Demonstrate the ability to collect specific data from clinicians or their staff on a clinically defined set of patients to enrich the EHR data set and answer effectiveness and safety questions concerning medical therapeutics. a. Demonstrate the ability of the federated system to use clinical and administrative data to identify patients from whom additional data might be collected. Technical Overview The Distributed Ambulatory Research in Therapeutics Network (DARTNet) builds on several best-in-breed technologies to create a true distributed clinical data repository for data acquisition and other activities. The system is currently based in primary care practices. It is not dependent on any particular electronic health record (EHR) for data access. Systems are in place to encourage high quality data collection: improved care processes (which increase the likelihood that selected data elements are captured), multiple data interfaces, data standardization, a data repository, and a GRID presentation for distributed query activities. Figure A-1 below summarizes the relationships between data sources and data access points. We will highlight elements of this figure throughout this document. A-4

5 Figure A-1. Relationship of data sources and the DARTNet architecture within a single organization DARTNet Translation Interface Other* EHR CDR CCR Gateway Research Portal POC NLP *Other data sources include billing data, hospital data, SureScripts data, and other third party databases. Web Services Queries and Data Transfers This document is organized from the point of contact between patient and physician during routine clinical care, moving toward use of the data for research and quality improvement. Beginning with the source of clinical data within the patient-centered medical home, this report will provide a technical description of the following critical data collection and processing components of DARTNet: Collecting EHR data from the primary care practice Interfaces to secure laboratory, radiology, and medication data Interfaces to secure hospital data Clinical decision support as the critical linkage between disparate EHR data and a centralized system Mapping EHR data Grid computing DARTNet queries (both local and global) Support for local quality improvement and large scale comparative effectiveness studies System security Natural language processing to obtain information entered as text Point-of-care data collection Clinical Data Sources The foundation of DARTNet lies in clinical data collected for patient care in the medical home. We rely on clinicians currently primary care clinicians who actively use an EHR to document ambulatory patient encounters. The system builds on two important features of practice-based research networks (PBRN) to enhance physician buy-in. First, the entire system is envisioned as a learning community for the member clinicians. The ability to connect to relevant clinical data allows the system to identify top performers so that the rest of the members can learn from them. Second, the system will develop point-of-care data collection processes, the A-5

6 sine qua non of a true PBRN. These points are also important as we believe both will drive improvements in the overall quality of data in each organization s EHR thus improving the quality of data available for research from DARTNet. In this section we outline the data DARTNet will acquire from these ambulatory EHRs and from interfaces with laboratory services, imaging providers, medication fulfillment services, and hospitals. Where possible, data that are not already flowing into the medical home for clinical use will be captured in such a way as to make it available for clinical purposes as well as quality improvement/research purposes. Electronic Health Record (EHR) Because DARTNet includes data acquisition as a key component of the system we have elected to work only with EHRs that include coded problem lists, electronic prescribing and laboratory interfaces. The EHR system must also allow read only access to a data extraction/standardization system. It is important to note that we believe virtually all EHRs that meet the minimum requirements above can be supported. EHRs that are known to be compatible with the current data extraction system include Allscripts, A4 Healthmatics, Centricity, EMDs, NextGen, Practice Partners, Soapnotes, and eclinical Works. DARTNet organizations must commit to using their EHRs in a way that will support the advanced clinical decision support system (CDSS), meaning that the organization uses limited locations for data elements and consistent terminology throughout the EHR. Where use is highly variable by member clinicians, the organization must develop a plan to improve data integrity to support the advanced clinical decision support system. Note that the minimum standards are couched in clinical decision support terms to help solve clinical problems facing DARTNet clinicians with the expectation that this will provide higher quality data for research, while also improving clinical care. See Appendix A-A for DARTNet membership criteria. Although EHRs are rich sources of data, they do have some limitations. Clinical data cannot be electronically extracted from scanned documents. Likewise it is very difficult to extract data from free text or electronic documents that contain embedded values (such as imaging reports). Thus, EHRs that rely heavily on, or implementations of EHRs that extensively use, free text, scanned text, or electronic interfaces of textual data (such as dictated notes) will be discouraged. A further limitation of using only clinically-derived data is that the frequency and consistency of data collection are likely to be less than ideal when compared to randomized controlled trials. It is important not to think of DARTNet as solely a data acquisition solution, but as a system that also supports randomized and practical clinical prospective trials. Data interfaces for DARTNet organizations are described below. All are interfaces that operate for clinical purposes. Some are required (and will be noted as such) and some are optional. The development of these interfaces generally occurs at the EHR vendor - data provider level (and occasionally at the CINA - data provider level). These interfaces are not maintained by the DARTNet infrastructure. They are described here for the reader to understand how clinical data are available for use by DARTNet. Laboratory, Imaging, and Medication Fulfillment Interfaces An electronic laboratory interface must be in place for the primary laboratory used by each organization, and preferably for all laboratories used by the organization. Electronic interfaces for an organization s secondary labs can be created to store data directly in the CDR if an EHR interface is not available. These labs would then support the CDSS/data extraction A-6

7 process. CINA creates a clinical data repository (CDR) at each organization (see Figure A-1, above) and the CDR can handle electronic interfaces instead of the EHR if this is more cost effective. Electronic interfaces with the imaging centers used by the organization are preferred but not required. Even electronically transmitted imaging data arrives as a text file and therefore requires processing for data extraction. SureScripts-RxHub is a partner in the project and all DARTNet sites are encouraged to install SureScripts-RxHub capabilities as the organization s EHR and region allow. These activities are part of the EHR installation and not within the control of the DARTNet infrastructure development. Once SureScripts-RxHub capabilities are installed, medication fulfillment data can flow either into the organization s EHR (preferred method) or into the CDR until the EHR can support these data. In either case, the medication fulfillment data are available for CDSS/QI/research purposes. Hospital Inpatient Data Interfaces Hospital inpatient data interfaces are not included in most ambulatory EHRs. In EHRs where inpatient data are included, the enormous volume of inpatient data has created difficulties in finding useful ways to sort inpatient and outpatient data for efficient review. Selected inpatient data elements can be useful, but the extent of the data required for outpatient clinical care is markedly less than the needs during many inpatient admissions. Therefore, we are exploring specific data interfaces for hospital data that would feed into the clinical data repository (CDR) and be accessed for quality improvement and research. At this time we are focusing on collecting data that will inform us of severe adverse events that originated in the ambulatory arena, rather than developing a robust inpatient related distributed database. Such interfaces require development with each institution in terms of data elements, frequency of data transfers and local data storage. Since DARTNet is also designed to support the patient-centered medical home, data extraction and storage of identified data (such that it can be linked to outpatient data) is HIPAA compliant as it will be available for clinical care. Demographic and Billing Data The ability to link to most billing and accounts receivable systems is a feature of the CDR in use for DARTNet. In general, the data contained in these systems are also included in the EHR. Therefore, at this time, interfaces to these data bases are not required of DARTNet participants, though some already have these interfaces in place. Clinical Decision Support and Data Extraction Clinical Integration Networks of America, Inc. (CINA) is a small corporation dedicated to providing an advanced clinical decision support system (CDSS) independent of a particular EHR. Practice-level implementation of the CDSS provides an avenue to communicate with clinicians at the point of care. DARTNet capitalizes on the CDSS to facilitate bi-directional communication with clinicians to collect EHR data from their practices, and to provide local clinical decision support services for clinicians to use for their own quality improvement initiatives. This communication can be tailored for each patient visit based on analysis of many different data elements from various sources, including the EHR, the billing system, direct patient data entry, drug fulfillment data and laboratory data that is not linked to the EHR. Additionally, the CDSS can standardize the EHR data elements that DARTNet will aggregate. A-7

8 Thus, the CINA software will populate the federated database, and provide a method through which DARTNet can manage point-of-care data collection. Using an Open Database Connectivity (ODBC) connection, CINA connects at the data level to many ambulatory EHRs. At the data level, CINA already has proven interfaces with many of the major ambulatory EHR products in the country as noted above. For this project the primary data sources will be each organization s EHR, augmented where necessary by billing data and medication fulfillment data at some locations, as described above. CINA completes data extraction and connectivity through a software package called the CINA Protocol Engine (PE). This software system runs locally on each practice s CINA server. The PE is programmed in C#.Net and is a DLL that is compiled through a table driven development process. This approach allows the medical staff at CINA to create the operators necessary to provide the CDSS function without having to be C# programmers. Through connectivity to each organization s EHR and ancillary databases the PE fills the CINA Clinical Data Repository (CDR). After the CDR is filled or appended the PE then performs data standardization and runs a series of CDSS functions (including research specific algorithms) against each patient for which new information has been appended to the CDR since the previous update. All outputs from the CDSS function (such as the need for a particular lab test or the need for additional data collection for a research project) are stored in the CDR and reported to clinicians at the time of the next patient visit and can be exported for batch usage (such as recall letters) and reporting activities. If there is a change in one of the protocols in the protocol engine then the system will run the new protocol against all eligible patients updating their CDSS results. On a timed, automatically triggered basis - typically early each morning the CINA system within each organization checks with the central CINA server to see if any protocol changes have been made to the organization s PE. If so, the local PE is updated and the new or changed protocols run against the CDR. Thus, we are able to rapidly implement new CDSS and research functions. The CINA CDR is a normalized and standardized database of relevant clinical information (not an image of the entire EHR database). The CINA CDR can be deployed in a number of ODBC compliant databases, but DARTNet uses a Microsoft Sequel deployment. The CDR stores the raw data format for each data element (in some cases data elements may be represented in the EHR in many different ways) and all outputs related to CDSS protocols are also stored. This allows us to recode any data element in the CDR if we discover a need to present the information in a more granular fashion. For instance, we are initially coding all Hemoglobin A1c values with one SNOMED CT code, though some tests are run in a central laboratory and some are performed using an analyzer at the point of care. If in the future it becomes important to tell the difference between these two circumstances we can look back into the CDR data and recode for each. Finally, within the CDR tags are placed on the data such that it can be appropriately loaded into Qlikview, a reporting structure that is included with the PE. Qlikview reports are used for the quality improvement/learning community activities. The point-of-care output of the CDSS report can be displayed on a web page, embedded within an EHR, or printed for use. All DARTNet practices have found that printing the output is a superior approach, as the information may be acted upon by the front desk staff, the patient, a nurse and the clinician. Paper is an easy and proven way to move the information between this diverse set of individuals during an office encounter. Also, several practices use the system to highlight patient education and support services on web sites based on diagnoses or other clinical A-8

9 data. In these practices the paper form is given to patients for educational and personal clinical information tracking purposes. Data Mapping With Each EHR Data elements identified as applicable to the CDSS process or for current or future studies are mapped within each EHR and standardized prior to being imported into the onsite CDR. The labor-intensive process of mapping has included identifying the variations on data content and storage locations occurring within each EHR and deciding how to apply standardized nomenclature to each data variant. The EHR mapping process (which uses the CINA Mapper) utilizes pattern matching to locate, verify and translate both codes and text into the standardized nomenclature maintained within the CDR. Whether data are stored in constrained fields or not, the CINA Mapper allows likely matches to be viewed by a clinician who then determines whether the match is congruent with the concept desired. Once congruence is established, the CDR Mapper will extract these data elements on a regular (typically nightly) basis from the EHR to the CDR. Data Standardization The CDR is primarily populated with data elements used for CDSS. All data elements in the CDR are standardized (cross-walked) to one of several coding systems; ICD-9 CM for diagnoses, RxNORM and GCN codes for drugs and SNOMED CT codes for all other data elements. The DARTNet Research Core has identified a robust initial set of data elements for standardization. Most elements have been cross walked to one of the above coding systems, and all data elements required for the pilot research project have been cross-walked (see Appendix A-B). SNOMED CT codes for all data elements including laboratory tests, selected imaging studies and procedures, history, allergies, and family history have been mapped and reviewed by SNOMED SAS, the training and certifying organization for SNOMED CT operated by the College of American Pathologists. We can also cross-walk the SNOMED-CT codes to LOINC for lab results if this is deemed appropriate in the future. Medication cross-walking poses the greatest challenge since information is needed both at the drug classification and the individual medication level and medications come as single entities and combination drugs. We have coded all single agent medications of interest for general CDSS use and for our first research project. We are continuing to explore the best use of SNOMED CT versus commercially available drug codes for group classifications and combination drugs, neither of which is currently included in RxNORM. As there is no single coding system that is comprehensive and without disadvantages, we are currently planning to cross-walk medications to both RxNorm and GCN codes as well as capturing NDC codes when available. We also must incorporate the ability to link both prescription generation data from the EHR to dispensing information from SureScripts- RxHub. As new data elements are added, or if greater detail is required from currently crosswalked codes, new entries will be added to the data dictionary and the CDR will be populated with these codes. All codes are reviewed by in-house physicians, our in-house coding expert, and intermittently by external coding consultants. Grid Based Computing This section of the report describes the computing resources needed to administer and run DARTNet. DARTNet builds upon the work previously carried out by the University of A-9

10 Minnesota (with funding from NIH) to build the epcrn Portal. This portal provides the architecture that houses the Gateway database for all users (local and centralized), as well as the query capabilities and the security systems. DARTNet uses the connectivity and distributed query capabilities of the epcrn Portal, which are handled through a specifically designed application of the Globus Toolkit including the X.509 security application, the Open Grid Services Architecture - Data Access and Integration (OGSA-DAI) application, and Globus File Transfer Protocol (FTP) applications. Grid computing is distinguished from conventional distributed computing by its focus on large-scale resource sharing, innovative applications, and in some cases, high-performance orientation. Grid computing is distinct from Internet access (which is primarily a communication tool) or wide area networks or virtual private networks, which allow access rights behind a fire wall. Grid computing allows functions such as complex queries to be passed to any number of local nodes without crossing an organization s firewall to physically access the data to be acted upon. The query can be executed locally and the output can either be stored on a local computer or, if allowed, returned to the central location. Since the Grid utilizes parallel processing, queries that may take hours to run against a central database can often be completed in minutes across a Grid system. For more details of Globus Toolkit and Grid computing, and their significance for such applications, see Appendix A-C. The basic structure of the DARTNet system (incorporating the CINA CDSS, epcrn Gateway Portal, and linkages to local EHRs) is displayed in Figure A-2. Each DARTNet organization has (1) an EHR database that is designed for clinical transactions and under the control of the local organization and EHR vendor, (2) a CDR database that is controlled by CINA (a HIPAA business associate of each organization) and the local organization and (3) a Grid-enabled database, called the epcrn Gateway database, that is controlled locally but accessed by the Technical Core of DARTNet. Figure A-2 displays the potential data feeds and the relationship of the various components of the system to each other. This model does not show all potential connecting arrows or copies of each database to simplify the diagram, but highlights multiple clinical organizations within the network. A-10

11 Figure A-2. Basic structure of the DARTNet system (incorporating the CINA CDSS, epcrn Gateway Portal, and linkages to local EHRs) EHR Rx Billing Billing Other EHR Rx Univ of MN CLINICAL DATA SYSTEMS (Labs, Billing) Internet CINA CDR Exports CCR QUALITY IMPROVEMENT Audit and Feedback Benchmarks Reminder Systems CINA CDR Exports CCR RSA dual authentication Clinical Trial Website Health System 1 Gateway Health System 2 Gateway CLINIC 1 Gateway CLINIC 2 Gateway CLINIC 3 Gateway Trial bank Clinicaltrials.gov OGSA-DAI Tools Security, authentication, linkage, manage processes IRB Evaluation RESEARCH Trial design Ontology tools for standardization Project approval Eligibility and recruitment Trial management DARTNet/ePCRN PORTAL QUALITY IMPROVEMENT Audit and Feedback Benchmarks Reminder Systems Data Export and Presentation to the Grid The movement of data from the CDR to the epcrn Gateway database is based on the ASTM standardized Continuity of Care Record (CCR). The CINA tools access data in the EHR, standardize the data, and then create CCRs locally for each eligible patient at each of the clinical organizations. CCR files consist of an XML string, which is passed to the epcrn Gateway database (created in MySQL), and the file is parsed into fields that are selectively available to outside Grid enabled queries, effectively de-identifying the dataset. The use of the CCR and XML allows all potential data elements to be exposed to a Grid query to be accounted for within the XML schema. Thus, the OGSA-DAI query engine does not have to be modified as new data elements are added to the Gateway database. Security Overview/ Authentication The overall DARTNet security model adopts a defense-in-depth strategy developed by the University of Minnesota for the epcrn Portal. We describe each piece in turn. The term clinic represents any holder of patient data that will be securely exposed to the DARTNet infrastructure. The DARTNet security framework is based on a X.509 Public Key Infrastruturebased (PKI) security scheme. The system supports mutual authentication between clients and services, Transport Level Security (TLS) based secure communication, and authorization (access control). For secured communication, both the server and the client must have a certificate and key and a proxy for TLS based communication. During the handshaking process, proxy certificates are exchanged as well as a client/server s public key to test each party s authenticity. A-11

12 If the handshake succeeds, a TLS-based secure session is set up and all Simple Object Access Protocol (SOAP) A-1 messages are encrypted and transferred securely. For access control to services, data resources, and database activities, each user is associated with a specific and identifiable DARTNet role (e.g. Admin, Research staff, Physician, Nurse, etc.) who is allocated a set of privileges that are controlled by the local clinic site. There are three different tiers for authorization and access control: (1) user-role mapping after authentication, (2) access control on resources and activities, and (3) access control on the database layer, as shown in Figure A-3 and Figure A-4. The current DARTNet implementation focuses on access control on the database layer. Figure A-3. Access control at the server side Figure A-4. epcrn security infrastructure Each clinical site is an independent research domain and is primarily responsible for local data security. The security requirements for the clinic sites consist of two types of controls: physical access control and technical security control. 1) Physical access control a. Physical access to computers and software systems (OGSA-DAI server and Database) is restricted. b. All monitors have a pre-defined time-out feature. c. Passwords for the database and the computers are properly and securely managed, including hard password requirements. 2) Technical security control a. Firewall setting for access control i. Only the Internet Protocol (IP) address of the DARTNet portal through the correct port number is allowed to access the OGSA-DAI server system. ii. Access to the MySQL database is only allowed over local connections. iii. All communication between the DARTNet portal and OGSA-DAI server are encryption based. b. SQL Query restriction A-1 A-12

13 i. Only SQL query (SELECT) is allowed from the Web Portal to the local Gateway database. ii. Any other SQL statements that might modify the database are deleted before being sent to the database. iii. SELECT statements that include a locally restricted column, such as patient identification information, are blocked by the OGSA-DAI server. c. Client/Server verification: Client/server verification (authentication) is performed based on x.509 based PKI infrastructure. i. Only the clients that have certificates legitimately signed by the DARTNet certificate authority are able to access the OGSA-DAI server. ii. Certificates are generated based on the RSA public-key authentication algorithm (a user held system that changes the authentication code between user and the OGSA-DAI server every 60 seconds.) d. Universal Description Discovery and Integration (UDDI) Server protection i. HTTPS protocols are used for all connections to the UDDI server in the DARTNet portal. The connection to the database (PostgreSQL) is restricted to local access only. Finally, where possible the entire system operates over Internet 2, thus further limiting potential data snooping. Data Query The DARTNet query architecture is shown in Figure A-5. The figure depicts the interactions such as registering, discovering, and querying databases in the clinics, which can be accessed via Grid services interfaces. A middleware, OGSA-DAI (Open Grid Service Architecture Data Access & Integration A-2 ), is used to allow data resources, such as relational or XML databases, to be exposed as Grid services. This OGSA-DAI data service is deployed in Figure A5. DARTNet query architecture A-2 A-13

14 the default Globus Toolkit Web Services Resource Framework (WSRF)-compliant Web Services Core java container. A-3 In order to support a Web-based distributed directory service (publishing/discovering Web services) for the OGSA-DAI services, we adopt Apache juddi, A-4 an open source Java implementation of UDDI specification. We have developed client Application Interfaces (APIs) supporting juddi publish/query, concurrent distributed queries, and error/exception handling. Query Development Queries are dynamically created by the epcrn Research Portal application. This is a web based interface that allows complex queries to be developed through an intuitive interface. The interface performs several functions: a. References the National Cancer Institute Enterprise Vocabulary Services (EVS) b. Allows queries to be generated with different code types c. Generates the executable query in the form of an XPath Query DARTNet will employ this interface in carrying out the research project designed to test the prototype system. The first study will be a comparative effectiveness study of oral hypoglycemic drugs. Query Execution Once a query has been developed the epcrn Research Portal application submits the query to the OGSA-DAI APIs, which pass them to each node within DARTNet to be run against the Gateway database through a Java application on the local server. All queries run locally and simultaneously. If the query is designed to return only aggregate data the results are returned in the same session to the OGSA-DAI server. If the results return de-identified patient level data the query can only be passed into the local server and must be activated by someone behind the local firewall. Results of these queries are returned locally and then, after local permission is given, transferred to the research team. These two additional steps guarantee local control over any patient-level data used for research purposes. Data Transfer Aggregate practice-level data are currently being transferred using two different mechanisms, with the expectation that eventually the Globus system will handle all data transfers. While the full Globus installation at each organization is being finalized and finetuned, we will use a secure FTP transfer from each location to a University of Colorado Department of Family Medicine secure server. We will use standard secure FTP software for this transfer and will set up the receiving server to transfer received files every 5 minutes to a data server, which is not visible to the Internet. As the system matures, we will be able to use the OGSA-DAI queries to return aggregate data directly from the organizational-level Gateway databases to the University of Minnesota (UMN). We can then aggregate further and move the data between the UMN and other research sites using either the Globus transfer capabilities or secure FTP. A-3 A-4 ws.apache.org/juddi A-14

15 Patient level data will always be extracted to a local directory within each organization. If the select statement is passed over the Globus system it must be activated by clinic personnel or a business partner. Clinic personnel must also manually move the data to a location that is accessible to the Globus secure transport functions. If the data extraction occurs directly against the CDR (as it does currently) it must be physically run by CINA personnel or a practice representative (these queries would be written by CINA using a stored procedure to create the data set). This data set can then be transferred through secure FTP. This redundant approach is more labor intensive but provides a backup data extraction system as we fine-tune the Globus environment. Quality Improvement For member organizations, one of the key DARTNet functions is the creation of an effective learning community. To this end, DARTNet s Board of Directors has directed the Technical Core to develop mechanisms to report both process and intermediate outcome quality measures at the practice level. The indicators must include both individual measures and combination measurements (such as an overall score for diabetes care in addition to presenting each measure independently). Finally, the Board instructed the Technical Core to explore appropriate mechanisms to control for variability in practice populations. Eventually, many of these quality improvement reports will be created by running a series of queries using the Gateway database. By varying the inclusion and exclusion criteria we should be able to explore the effects on quality indicators of a number of variables, such as the number of visits in a year, insurance status and other variables. While we are finalizing and testing the Gateway database and Grid services we will rely on existing quality reports within each organization to create these system-wide reports. The CINA system tags data as it populates the CDR and these tags allow the building of a sophisticated reporting system using the Business Intelligence software package Qlikview. This report allows an organization to move from the full organization level to individual patient level data in a few clicks. We truncate these data at the practice level and provide tags for additional population-level variables of interest so that the quality reports can take these variables into account. Paper copies of these reports are available as part of the Qlikview package at each organization and on a secure SharePoint web site. The SharePoint site uses Microsoft Sequel Reporting Services to create dynamic reports that DARTNet members can manipulate. Reports only identify a single practice or an organization s full set of practices, depending on an individual s level of permission on the SharePoint site. The three top-performing practices for each final report product (the report that best controls for population variability) will be identified so that other organizations may learn from them. Data Additions Through Natural Language Processing Even in EHRs that offer extensive data templates for coding history and physical exam items, these areas are frequently entered using free text because of the clinical nuances involved. Therefore, by using natural language processing (NLP), DARTNet data will be significantly enhanced for quality improvement and research purposes. The free text of EHRs has significant benefits for clinical research and care, as has been shown for heart failure (Pakhomov et al., 2007a) and chronic angina (Pakhomov et al., 2007b). EHR text constitutes a more complete source of information than billing records for identifying A-15

16 patients clinically relevant data. Even relatively simple NLP methods can be used to unlock the valuable patient- and disease-specific information from an EHR. This methodology may be used to gather data from clinical reports to improve the quality and safety of care and for research. For example, when searching for evidence of foot exams in patients with diabetes, the sensitivity of the NLP approach was 91% (95% CI 85-96), the specificity was 76% (95% CI 58-94), while the overall accuracy was 88% (95% CI 82-94). The reliability of manual review was 91% (95% CI: 85-97), which is not significantly different than using NLP. The DARTNet Natural Language Processing system will be designed initially for processing the text of physical examination, procedural or history of present illness data. The system will be constructed by adapting existing basic components to the task at hand and will rely on the publicly available software platform Unstructured Information Management Architecture (UIMA). A-5 UIMA was developed by IBM, Inc. and represents a powerful Javabased software platform for development and implementation of modularized applications for processing unstructured data including text, video, audio and genomic data. The UIMA Working Group was sponsored in 2005 by the US government. This working group facilitated the integration of several similar platforms including the Generalized Architecture for Text Engineering A-6 (GATE) and Stanford s OpenNLP toolkit. A-7 All of these platforms are publicly distributed as open source. For this project, we propose to use UIMA to implement the NLP system because it is emerging as a powerful, versatile, and well-documented technology that is widely accepted in the bio-nlp community. One of the unique advantages of UIMA is that it supports distributed grid-enabled applications, which is critical for the interface with the epcrn system. Furthermore, our group has prior experience with implementation of NLP technology using UIMA. The NLP system will consist of the components described below and illustrated in Figure A-6. Common Annotation Structure (CAS) Initializer: This is a pre-processor component specific to the UIMA platform. Its purpose is to provide an interface to the clinical report on epcrn servers and to generate a Common Annotation Structure. The latter is a Java data structure in object form that will be subsequently manipulated by the NLP components. Tokenizer: The function of the tokenizer is to break up the continuous stream of text into its basic segments including word tokens, digit tokens and punctuation tokens that are compatible with subsequent NLP components. Sentence Detector: The function of the sentence detector is to identify sentence boundaries in the tokenized stream of text to support subsequent processing. The sentence detector is based on an open-source Maximum Entropy A-8 classifier that categorizes punctuation tokens as either constituting a sentence boundary or not. For example, the period in Dr. Smith would be categorized as a non-boundary token, while the period in No skin lesions. will be classified as a sentence boundary. Part-of-speech (POS) Tagger: The function of the POS Tagger is to identify the appropriate lexical category (noun, verb, preposition, conjunction, etc.) from a list of 45 Penn A-5 A-6 A-7 A-8 A-16

17 Treebank part-of-speech categories. We propose to use a publicly available Hidden Markoff Model (HMM) based POS tagger Medpost. For example, the words in the sentence Patient has no pedal pulses will be tagged as Patient/NN has/aux no/det pedal/adj pulses/nns. The part-of-speech information is necessary for the subsequent step of syntactic parsing. Figure A-6. Natural language processing components high-level architecture Medical text (e.g. Patient has no pedal pulses. ) CAS Initializer Tokenizer Sentence Detector POS tagger Syntactic Parser CAS Consumer Modification identifier Concept Mapper Lucerne index UMLS Metathesaurus <concept cui=c mod=absent> Dorsalis Pedis </concept> Stanford Syntactic Parser: The function of the syntactic parser is to identify the syntactic structure and composition of phrases within sentences identified by the Sentence Detector component. For example, the sentence Patient/NN has/aux no/det pedal/adj pulses/nns. will be parsed as shown in Figure A-7. Figure A-7. Example of sentence parsing S NP VP NP Patient/NN has/aux no/det pedal/adj pulses/nns A-17

18 This simple parse tree represents two noun phrases ( patient and no pedal pulses ) and one verb phrase ( has no pedal pulses. ). This structured representation of the sentence will enable subsequent mapping of the sentence to the UMLS metathesaurus and identification of modifications such as negation. Concept Mapper: The purpose of the concept mapper is to assign unique identifiers to medical concepts identified within the phrases parsed in the previous step. The mapping proceeds by permuting the words within each phrase and then looking up each permutation in the UMLS Metathesaurus. The UMLS Metathesaurus is the largest collection of medical vocabularies maintained by the National Library of Medicine. The Metathesaurus contains over 100 medical vocabularies and encompasses over 1 million medical concepts. One of the valuable characteristics of the Metathesaurus is that it maintains a set of orthographic, lexical and semantic variants for each concept identifier (e.g. Babinski s Reflex = Babinski s Sign = Extensor Plantar Reflex aggregated under the Concept Unique Identifier C ). In addition to the richness of the Metathesaurus, the National Library of Medicine makes available a Lexical Variant Generator (LVG) tool specifically designed to identify orthographic and lexical variants of medical terms. We will use LVG as part of the mapping process. Modification Identifier: The function of this component is to look for modification of the concepts identified in the previous steps. Identifying modification of concepts is critical for information retrieval from clinical documents. In particular, identification of negated concepts (e.g. denies chest pain, absent pulses ) is crucial to constructing the correct representation of a physical examination from the text of the clinical report. We will use a publicly available tool, NegEx, specifically designed to identify negation in clinical discourse. NegEx is a regular expression mechanism with a set of rules that designate keywords indicating negation in the vicinity of the index term. Despite its simplicity, it has been shown to perform with 78% sensitivity and 94.5% specificity on terms identified in hospital discharge summaries. In addition to negation, we will generalize NegEx to determine other statuses of the findings including history and family history. This will be done by manual examination of a random sample of 500 finding mentions occurring in the electronic medical records to identify the adjustments and extensions necessary to adapt NegEx to the task at hand. Common Annotation Structure (CAS) Consumer: This UIMA platform component is responsible for converting the internal representation of the elements identified by the NLP components to XML format for input to the indexing engine. Lucene Index: Lucene A-9 is a high-quality open-source indexing engine specifically designed to enable fast indexing and search of textual documents. The engine is also capable of indexing non-textual fields including numerical values, thus making Lucene equivalent to a database for most common applications. One of Lucene s features that is not commonly available in commercial databases is the ability to restrict searches by proximity of elements found in the text. This feature makes Lucene particularly attractive for indexing medical records as it allows greater flexibility in constructing ad-hoc queries and will serve as a mechanism to overcome possible limitations of the NLP methods and the UMLS Metathesaurus. The medical records will be indexed both on the concepts identified by NLP and the standard single keywords normalized with LVG. A-9 A-18

19 The process illustrated in Figure A-7 will run in batch mode to process relevant records from the participating centers. While the UMLS is a rich ontology comprising a large number of different sources of medical terminology, the vocabularies that are part of the UMLS are not fully curated by the UMLS staff and are accepted as is leading to potential problems with granularity and possible inconsistencies. In order to alleviate this concern we will use a subset of the UMLS consisting of the following sources: Systematized Nomenclature of Medicine, Clinical Terminology (SNOMED-CT), International Classification of Disease (ICD-9, 10), Medical Subject Headings (MeSH) and the National Cancer Institute Thesaurus (NCI Thesaurus). Directed Point-of-Care Data Collection Even in a highly coded EHR there will be data elements that are essential to understanding the effectiveness or safety issues related to a therapeutic agent that are not likely to be included in a clinical note. Examples include a PHQ-9 score for all depressed patients or a standardized assessment of bipolar symptoms in patients starting to take an antidepressant. In these situations the ability to direct data collection during an office visit will enable the collection of additional critical data to supplement routine clinical data for selected patients. This ability will allow DARTNet to essentially create a controlled trial environment within the routine care process. The CINA PE creates a clinical decision support report for every patient visiting DARTNet practices. This report can also direct specific data collection at the level of a patient or data element, based on existing data within the CDR. This ability to fill in missing data and supplement clinical data is one of the reasons the DARTNet Board demanded that any CDSS system be able to support point-of-care recommendations. There are two possible scenarios to be dealt with: (1) the data are frequently collected during routine care but the results are missing on a large portion of the population of interest or (2) the data are not typically collected to the degree of standardization that is needed for the study in question. For the first scenario, where the data are often collected during routine care, the CINA PE would be programmed to look for these data within the specified period of time when a patient eligible for the study is being seen. If the data element is present and timely then no prompt would be included on the point-of-care report. If the data element is lacking, a request to collect the information would be generated. In the second scenario, where data standardization needs to be improved (such as a standardized assessment of depression severity) the CINA PE would be programmed to print the full data collection form for patients meeting study criteria. Depending on the questions to be asked and other factors this could be done only with patients who have provided their consent to participate, or could be done as an extension of clinical care. The results of this standardized assessment would then be entered into the EHR for extraction to the CDR and eventually to the study team. Summary The DARTNet data collection, standardization, presentation, query and reporting system is a state of the art implementation of a series of public and private software systems that have been linked to provide data acquisition, data standardization and quality improvement activities. The system is independent of most EHRs, can extract data from multiple data sources and A-19

20 supports centralized research activities as well as local and system-wide quality improvement and learning. A-20

21 References Pakhomov S, Weston SA, Jacobsen SJ, et al (2007a). Electronic medical records for clinical research: application to the identification of heart failure. American Journal of Managed Care 13(6 Part 1), Pakhomov SS, Hemingway H, Weston SA, et al (2007b). Epidemiology of angina pectoris: role of natural language processing of the medical record. American Heart Journal 153(4), A-21

22 Appendix A-A: Criteria for Adding New Member Organizations to DARTNet 1. Data extraction from electronic health record to DARTNet specifications a. Minimum data types must be included 2. Use advanced clinical decision support at point of care a. implementation process b. provider level 3. Organizational commitment to the process 4. Organizational key leaders engaged 5. Share de-identified aggregate data (with appropriate safeguards/approvals) 6. Willingness to be identified if top performer 7. Adds to research/learning needs of DARTNet A-22

23 Appendix A-B: Coding Dictionaries for DARTNet Data Procedures Full Specified Name Master Concept ID Concept ID Lymphocyte count (procedure) Neutrophil count (procedure) Alanine aminotransferase measurement (procedure) Albumin measurement (procedure) Alkaline phosphatase measurement (procedure) Aspartate aminotransferase measurement (procedure) Urine bacteria test (procedure) Bilirubin measurement (procedure) Bilirubin, direct measurement (procedure) Bilirubin, total measurement (procedure) Blood urea nitrogen measurement (procedure) C-reactive protein measurement (procedure) Calcium measurement (procedure) Carbon dioxide measurement (procedure) Urine microscopy for casts (procedure) Creatinine measurement (procedure) Erythrocyte sedimentation rate measurement (procedure) T4 free measurement (procedure) Glucose measurement, fasting (procedure) Glucose measurement, random (procedure) High density lipoprotein measurement (procedure) Hemoglobin determination (procedure) Hemoglobin A1c measurement (procedure) Hepatitis B core antibody level (procedure) Hepatitis B e antibody level (procedure) Hepatitis B e antigen test (procedure) Hepatitis B surface antibody measurement (procedure) Hepatitis B surface antigen measurement (procedure) Hepatitis C antibody measurement (procedure) Hepatitis C antigen measurement (procedure) Low density lipoprotein cholesterol measurement (procedure) Mean corpuscular hemoglobin determination (procedure) Mean corpuscular volume - (procedure) Microalbumin measurement, urine, quantitative (procedure) Platelet count (procedure) Potassium measurement (procedure) A-23

24 Glucose measurement, blood (procedure) Blood sodium measurement (procedure) Urine dipstick for specific gravity (procedure) Total cholesterol measurement (procedure) Total protein measurement (procedure) Triglycerides measurement (procedure) Thyroid stimulating hormone measurement (procedure) Urine dipstick for glucose (procedure) Urine dipstick for hemoglobin (procedure) Hemoglobin determination, urine (procedure) Urine dipstick for protein (procedure) White blood cell count (procedure) Urine dipstick for urobilinogen (procedure) Urine dipstick for ketones (procedure) Urine dipstick for nitrite (procedure) Urine dipstick for leukocyte esterase (procedure) Tri-iodothyronine measurement, total (procedure) Urobilinogen measurement, urine (procedure) Urine ketone test (procedure) Nitrite measurement (procedure) Leukocyte esterase measurement (procedure) Protein measurement, urine (procedure) Hemoglobin determination, urine (procedure) Urinalysis, specific gravity measurement (procedure) Platelet mean volume determination (procedure) Chlamydia trachomatis DNA assay (procedure) Urine protein/creatinine ratio measurement (procedure) Microalbuminuria measurement (procedure) Urine microalbumin/creatinine ratio measurement (procedure) Prostate specific antigen measurement (procedure) Influenza vaccination (procedure) Pneumococcal vaccination (procedure) Tetanus vaccination (procedure) Vaccination for diphtheria, pertussis, and tetanus (procedure) International normalized ratio (observable entity) Measurements Full Specified Name Master Concept ID Concept ID Body height measure Body mass index (observable entity) Systolic blood pressure (observable entity) A-24

25 Diastolic blood pressure (observable entity) Standing systolic blood pressure (observable entity) Standing diastolic blood pressure (observable entity) Sitting systolic blood pressure (observable entity) Sitting diastolic blood pressure (observable entity) Lying systolic blood pressure (observable entity) Lying diastolic blood pressure (observable entity) Body weight measure (observable entity) Body temperature Pulse Functional Tests Full Specified Name Master Concept ID Concept ID Forced expired volume in 1 second (observable entity) Total vital capacity measurement (procedure) Forced expired volume in one second/vital capacity ratio (observable entity) Maximum breathing capacity, function (observable entity) Peak expiratory flow rate (observable entity) Total vital capacity measurement (procedure) Electrocardiographic procedure (procedure) Holter extended electrocardiographic recording (regime/therapy) Hour ECG (procedure) Cardiovascular stress testing (procedure) Transesophageal echocardiographic monitoring features (observable entity) Exercise stress echocardiography (procedure) Transthoracic echocardiography (procedure) Echocardiography (procedure) Forced expiratory flow rate between 25+75% of vital capacity (observable entity) Family and Personal Medical History Full Specified Name Master Concept ID Concept ID Family history of malignant neoplasm of breast Family history of cancer of colon (situation) Family history: premature coronary heart disease (situation) Family history: Diabetes mellitus (situation) [D]Nonspecific abnormal Papanicolaou cervical smear NOS (situation) H/O splenectomy A-25

26 Functional asplenia Family history: Asthma (situation) History of - asthma (situation) History of - atrial fibrillation (situation) Personal history of primary malignant neoplasm of breast (situation) History of - cardiovascular disease (situation) Family history: Cardiovascular disease (situation) History of malignant neoplasm of cervix (situation) Family history: neoplasm of cervix (situation) History of - renal failure (situation) History of malignant neoplasm of colon Family history of cancer of colon (situation) Family history of chronic obstructive lung disease (situation) History of - chronic obstructive airway disease (situation) History of - diabetes mellitus (situation) History of - liver disease (situation) Family history: Liver disease (situation) History of - hypercholesterolemia (situation) Family history: Hypercholesterolemia (situation) History of - hypertension (situation) Family history: Hypertension (situation) Premature menopause (finding) Metabolic syndrome X (disorder) History of - myocardial infarction (situation) Family history: Myocardial infarction (situation) History of - kidney disease (situation) Family history of kidney disease (situation) History of - obesity (situation) Family history: Obesity (situation) Osteopenia (disorder) History of - pregnancy (situation) History of malignant neoplasm of prostate (situation) Family history of prostate cancer (situation) Family history: Sickle cell anemia (situation) Sickle cell anemia NOS (disorder) History of - hormone replacement (HRT) (situation) Osteoporosis (disorder) Family history: Osteoporosis (situation) Ex-smoker (finding) Non-smoker (finding) A-26

27 Smoker (finding) Procedure History Full Specified Name Master Concept ID Concept ID History of colectomy (situation) History of - renal dialysis (situation) History of - hysterectomy (situation) History of - lower limb amputation (situation) History of bilateral mastectomy (situation) Bone density scan (procedure) Colonoscopy (procedure) Seen in diabetic eye clinic (finding) Diabetic foot examination (regime/therapy) Double contrast barium enema (procedure) Fecal occult blood test (procedure) Flexible fiberoptic sigmoidoscopy (procedure) Date of last mammogram Papanicolaou smear test (procedure) Medication Allergies Full Specified Name Master Concept ID Concept ID History of - angiotensin II receptor antagonist allergy Angiotensin-converting-enzyme inhibitor allergy Adrenergic neurone blocking drug allergy Acarbose allergy Insulin allergy Sulfonylurea allergy Adrenergic neurone blocking drug allergy Beta-adrenoceptor blocking drug allergy Calcium-channel blocker allergy Digoxin allergy Diuretic allergy HMG COA reductase inhibitor allergy Lipid-lowering drug allergy Nicotinic acid allergy Omega 3-marine triglycerides allergy Biphosphonates allergy Anticoagulant allergy Salicylate allergy Warfarin allergy (disorder) Biguanide allergy A-27

28 Influenza vaccine allergy Pneumococcal vaccine allergy (disorder) Tetanus vaccine allergy (disorder) Medications Medication Name RxNorm Code GCN Code Abacavir Acarbose Acarbose Aceclofenac Acemetacin Acetaminophen Acetazolamide Acetohexamide Allopurinol Amiodarone Amitriptyline Amoxapine Clavulanate Amoxicillin Apazone Aspirin Atorvastatin Azathioprine Benazepril Benorilate Black Cohosh Extract Black Cohosh Root Extract Bosentan Bromfenac Bumetanide Butriptyline Candesartan Captopril Carbamazepine Carprofen Celecoxib Celecoxib Cerivastatin Sodium Chlorpromazine Chlorpropamide A-28

29 Chlorthalidone Citalopram Clomipramine Clopamide Comfrey Preparation Cyclophosphamide Cyclosporine Desipramine Dibenzepin Dichlorphenamide Diclofenac Didanosine Diflunisal Diltiazem Disulfiram Dothiepin Hydrochloride Doxepin Emtricitabine Enalapril Eprosartan Erythromycin Erythromycin Estolate Erythromycin Ethylsuccinate Erythromycin Gluceptate Erythromycin Lactobionate Erythromycin Stearate Erythromycin Stinoprate Escitalopram Oxalate Estrogens 4100 Etodolac Etoricoxib Exenatide Ezetimibe Fatty Acids, Omega Felbamate Fenbufen Fenofibrate Fenoprofen Fluoxetine Flurbiprofen Fluvastatin A-29

30 Fluvoxamine Maleate Fosinopril Garlic Preparation Gemfibrozil Glimepiride Glipizide Glyburide Glyburide, Micronized Greater Celandine Ibuprofen Imipramine Indapamide Indomethacin Iprindole Irbesartan Isoniazid Kava Root Kava Preparation Ketoconazole Ketoprofen Ketorolac Lamivudine Lisinopril Lofepramine Lornoxicam Losartan Lovastatin Loxoprofen Mafenide Magnesium Salicylate Meclofenamic Acid Mefenamic Acid Mefruside Meloxicam Dipyrone Metformin Methimazole Methotrexate Methyl Salicylate Methyldopa Metolazone A-30

31 Miglitol Minocycline Nabumetone Naproxen Nateglinide Niacin Nimesulide Nitrofurantoin Nortriptyline Olmesartan Medoxomil Opipramol Oxaprozin Oxyphenbutazone Parecoxib Paroxetine Pemoline Perindopril Phenobarbital Phenylbutazone Phenytoin Pioglitazone Piroxicam Pramlintide Pravastatin Probenecid Procainamide Propylthiouracil Protriptyline Pyrazinamide Quinapril Quinidine Ramipril Ranitidine Repaglinide Rifampin Rofecoxib Rosiglitazone Rosuvastatin Salsalate Sertraline Simvastatin A-31

32 Sitagliptin Skullcap Preparation St. John s Wort Extract Stavudine Sulfacetamide Sulfadimethoxine Sulfamethoxazole Sulfanilamide Sulfasalazine Sulfinpyrazone Sulindac Sulthiame Sumatriptan Suprofen Tacrine Tamoxifen Telmisartan Tenoxicam Terbinafine Tiaprofenic acid Tolazamide Tolbutamide Tolcapone Tolmetin Trazodone Trimipramine Troglitazone Troglitazone Trovafloxacin Valdecoxib Valerian Root Valproic Acid Valsartan Vitamin A Xipamide Zafirlukast Zalcitabine Zidovudine Zileuton Zofenopril A-32

33 Appendix A-C: About the Globus Toolkit The Globus toolkit is an enabling technology that has become the de-facto standard for technology solutions that enable secure sharing of databases across organizational boundaries. The federal government has recognized the significance of the Globus Framework, and has adopted it for several key applications. This includes the use of use of the Globus framework for cabig, which is NCI s Cancer Bioinformatics Grid. Globus is made universally available to qualified groups, which also makes it ideally suited for use in large-scale applications where replicability is critically important. The epcrn Portal makes use of the Globus technology for these reasons. The information in Appendix A-C provides a detailed overview of Globus as background for why this technology was chosen for the development of the epcrn Portal (developed by the University if Minnesota), and the relative importance of this technology to the University of Colorado in selecting the epcrn Portal as a key component of the DARTNet system. Grid Technologies Grid computing is distinguished from conventional distributed computing by its focus on large-scale resource sharing, innovative applications, and, in some cases, highperformance orientation. Grid computing facilitates flexible, secure, coordinated resource sharing among dynamic collections of individuals, institutions, and resources. In such settings, one encounters unique authentication, authorization, resource access, resource discovery, and other challenges. It is this class of problem that is addressed by Grid technologies. The real and specific problem that underlies the Grid concept is coordinated resource sharing and problem solving in dynamic, multi-institutional virtual organizations. This sharing is not primarily file exchange but rather direct access to computers, software, data, and other resources, as is required by a range of collaborative problem-solving and resource brokering strategies emerging in industry, science, and engineering. This sharing is, necessarily, highly controlled, with resource providers and consumers defining clearly and carefully just what is shared, who is allowed to share, and the conditions under which sharing occurs. A set of individuals and/or institutions defined by such sharing rules form what we call a virtual organization. Current distributed computing technologies do not address the concerns and requirements just listed. For example, current Internet technologies address communication and information exchange among computers but do not provide integrated approaches to the coordinated use of resources at multiple sites for computation. Business-to-business exchanges focus on information sharing (often via centralized servers). So do virtual enterprise technologies, although here sharing may eventually extend to applications and physical devices. Enterprise distributed computing technologies such as CORBA and Enterprise Java enable resource sharing within a single organization. The Open Group s Distributed Computing Environment supports secure resource sharing across sites, but most virtual organizations would find it too burdensome and inflexible. Storage service providers and A-33

34 application service providers allow organizations to outsource storage and computing requirements to other parties, but only in constrained ways; for example, storage service providers resources are typically linked to a customer via a virtual private network (VPN). Emerging Distributed computing companies seek to harness idle computers on an international scale but, to date support only highly centralized access to those resources. In summary, current technology either does not accommodate the range of resource types or does not provide the flexibility and control on sharing relationships needed to establish virtual organizations. Because of their focus on dynamic, cross-organizational sharing, Grid technologies complement rather than compete with existing distributed computing technologies. For example, enterprise distributed computing systems can use Grid technologies to achieve resource sharing across institutional boundaries; in the storage service providers and application service providers space, Grid technologies can be used to establish dynamic markets for computing and storage resources, hence overcoming the limitations of current static configurations. Effective virtual organization operation requires that we be able to establish sharing relationships among any potential participants. Interoperability is thus the central issue to be addressed. In a networked environment, interoperability means common protocols. Hence, Grid architecture is first and foremost a protocol architecture, with protocols defining the basic mechanisms by which virtual organization users and resources negotiate, establish, manage, and exploit sharing relationships. A standardsbased open architecture facilitates extensibility, interoperability, portability, and code sharing; standard protocols make it easy to define standard services that provide enhanced capabilities. We can also construct Application Programming Interfaces and Software Development Kits to provide the programming abstractions required to create a usable Grid. Together, this technology and architecture constitute what is often termed middleware ( the services needed to support a common set of applications in a distributed network environment ). The following brief and partial list provides a resource specific characterization of capabilities. Computational resources: Mechanisms are required for starting programs and for monitoring and controlling the execution of the resulting processes. Management mechanisms that allow control over the resources allocated to processes are useful, as are advance reservation mechanisms. Inquiry functions are needed for determining hardware and software characteristics as well as relevant state information such as current load and queue state in the case of scheduler-managed resources. Storage resources: Mechanisms are required for putting and getting files. Thirdparty and high-performance (e.g., striped) transfers are useful, as are mechanisms for reading and writing subsets of a file or executing remote data selection or reduction functions. Management mechanisms that allow control over the resources allocated to data transfers (space, disk bandwidth, network bandwidth, CPU) are useful, as are advance reservation mechanisms. Inquiry functions are needed for determining hardware and software characteristics as well as relevant load information such as available space and bandwidth utilization. Network resources: Management mechanisms that provide control over the resources allocated to network transfers (e.g., prioritization, reservation) can be A-34

35 useful. Inquiry functions should be provided to determine network characteristics and load. Code repositories: This specialized form of storage resource requires mechanisms for managing versioned source and object code; for example, a control system such as CVS. Catalogs: This specialized form of storage resource requires mechanisms for implementing catalog query and update operations: for example, a relational database. The Globus Toolkit has emerged as the dominant middleware for Grid deployments worldwide. Globus Toolkit The open source Globus Toolkit is a fundamental enabling technology for the Grid, letting people share computing power, databases, and other tools securely online across corporate, institutional, and geographic boundaries without sacrificing local autonomy. The toolkit includes software services and libraries for resource monitoring, discovery, and management, plus security and file management. In addition to being a central part of science and engineering projects that total nearly a half-billion dollars internationally, the Globus Toolkit is a substrate on which leading IT companies are building significant commercial Grid products. The toolkit includes software for security, information infrastructure, resource management, data management, communication, fault detection, and portability. It is packaged as a set of components that can be used either independently or together to develop applications. Every organization has unique modes of operation, and collaboration between multiple organizations is hindered by incompatibility of resources such as data archives, computers, and networks. The Globus Toolkit was conceived to remove obstacles that prevent seamless collaboration. Its core services, interfaces and protocols allow users to access remote resources as if they were located within their own machine room while simultaneously preserving local control over who can use resources and when. The Globus Toolkit has grown through an open-source strategy similar to the Linux operating system, and distinct from proprietary attempts at resource-sharing software. This encourages broader, more rapid adoption and leads to greater technical innovation, as the open-source community provides continual enhancements to the product. The Globus Toolkit has been designed to use (primarily) existing fabric components, including vendor-supplied protocols and interfaces. However, if a vendor does not provide the necessary Fabric-level behavior, the Globus Toolkit includes the missing functionality. For example, inquiry software is provided for discovering structure and state information for various common resource types, such as computers (e.g., operating system version, hardware configuration, load scheduler queue status), storage systems (e.g., available space), and networks (e.g., current and predicted future load) and for packaging this information in a form that facilitates the implementation of higherlevel protocols, specifically at the Resource layer. Resource management, on the other hand, is generally assumed to be the domain of local resource managers. A-35

36 Essential background is contained in the papers Anatomy of the Grid and Physiology of the Grid : The Anatomy of the Grid: Enabling Scalable Virtual Organizations. I. Foster, C. Kesselman, S. Tuecke. International J. Supercomputer Applications, 15(3), The Physiology of the Grid: An Open Grid Services Architecture for Distributed Systems Integration. I. Foster, C. Kesselman, J. Nick, S. Tuecke, Open Grid Service Infrastructure WG, Global Grid Forum, June 22, Figure A-8. Elements of the Globus Toolkit A-36