Data Quality of the Clinical Trial Process Costly Regulatory Compliance at the Expense of Scientific Proficiency

Transcription

1 Data Quality of the Clinical Trial Process Costly Regulatory Compliance at the Expense of Scientific Proficiency Mats Lörstad* Quality Technology and Management, Institute of Technology, Linköping University, Linköping, Sweden Summary The reasons for the belief in the need for perfection of clinical data are carefully nursed myths. The logical shortcomings in Good Clinical Practice are nothing but embarrassing to its scientifically trained promoters. Modern quality management methods, which increasingly degrade the importance of inspections, are described and their application in clinical research recommended. The documentation of procedures to attain acceptable quality levels of trial data is suggested to be standardized into a Quality Declaration. Copyright 2004 John Wiley & Sons, Ltd. Key Words quality management; good manufacturing practice; good clinical practice; quality assurance; quality declaration; process analytical technology Background: What s in the term Data Quality in Clinical Trials? It is talked about a lot, but there is no established definition of this quality. From how this undefined term is used in the reporting of clinical trials, however, it can be deduced that a favourite understanding is that the data gives an accurate *Correspondence to: Mats Lörstad, Quality Technology and Management, Institute of Technology, Linköping University, Linköping, Sweden. matlo@ikp.liu.se representation of what we measure to evaluate effects of the treatments under comparison. The clinical data recording process, however, as inspected by auditors is initiated with the recorded data, either in the Case Report Form, or in some other journal, or in an electronic database. From there, the data subjected to the process of data management is repeatedly inspected to ensure that the data that finally appear in the report are identical to the original, i.e. nothing untoward has happened on the way. This interpretation of quality does not consider the initial, preparatory work necessary to ensure that the measurement methods used are standardized, reliable and valid. Furthermore, it does not consider whether they are applied with the same rigour to a carefully made selection of relevant variables by all investigators contributing to the database. Who Defines Appropriate Levels of Data Quality? The critical case is the final, crucial phase of clinical research: phase III, in which the pharmaceutical firm is obliged to provide evidence that the new treatment is as good and safe as it is purported to be. From data quality point of view this raises many tricky problems: huge amounts of data and many persons involved working under very tight deadlines. In addition to this, the sponsor attempts perfection in all details of the study procedures. Quality Assurance Quality Assurance is defined as: All those planned and systematic actions that are DOI: /qaj.288

2 178 M. Lörstad established to ensure that the trial is performed and the data are generated, documented (recorded), and reported in compliance with good clinical practice (GCP) and the applicable regulatory requirement(s) [1]. It is a procedure carried out by GCP specialists who look at how the documentation specifies what needs to be done to ensure that the data, as originally recorded, will not be changed without documented reason by authorized persons in the data management processes. It does not discuss whether the information collected to evaluate the treatments under comparison is likely to become valid, reliable and accurate. Thus, its regulatory value provides the justification for its collection, certainly not its scientific value. Pharmaceutical GCP departments take a clean record from quality assurance as documented proof of quality. There is, however, no regulatory demand for perfection [2]. Yet another related issue: application of strict GCP methodology does not assure that such a noble goal is reached. One may wonder what nurses the growth of the quality assurance bureaucracy. Do the GxP Systems Assure High Quality? The origin of other GxP systems is good manufacturing practice (GMP). It was developed to ensure that the processes used in the manufacturing of drugs are harmonized with the goal of high quality. It emphasises the need for prevention of errors since the effect of errors on the large-scale production of drugs is highly costly. From costly experience the training factor has become very important. Yet, there are substantial resources spent on inspections, internal as well as external. These inspections seem to be carried out for regulatory reasons. To fulfil their responsibility for product quality, quality departments are growing to half of the size of the production departments. Does that appear to be a reasonable relation? Also good laboratory practice (GLP) is a predecessor to GCP with much more of the same characteristics as GMP. For convenience, the following discussion refers to GMP only. Since GCP is modelled upon GMP, and is referred to as a quality system, we are stimulated to believe that GCP is a stamp of good quality. There are, however, fundamental differences in the philosophy as well as in the coverage of these two systems. These differences have a direct impact upon their designation as quality systems. From the perspective of Quality Management terminology, this characterization is a misnomer: neither GMP nor GCP are quality systems; they are systems aimed at ensuring that the documentation describing the processes is complete and up-to-date. In what way are the GMP and GCP Systems Different? The main difference between GCP and GMP is the emphasis on training. GMP sets hard and comprehensive rules for the competence of the operators in drug production. No person is allowed to have anything to do with the production of the product unless they have passed the examinations in regularly repeated and up-dated training courses. Absence of an up-dated training record is taken as proof of insufficient training. In GCP, the need for training is focussed on the sponsor staff. The issue of training of the investigator is barely mentioned and then only in connection with education and experience, i.e. of historic value for the CV. No specific training is required for each actual study with its own data collection methods. Thus, even if the investigator is the main producer of data, his/her ability to deliver accurate data is taken for granted. He/she is not requested to undergo any training courses at all unless electronic CRFs are used for data recording. Shortcomings of GCP A most serious shortcoming of GCP, i.e. ICH GCP, is that it does not cover the entire sequence of procedures for clinical trials. In other words, it is incomplete. The coverage of GCP does not start

3 Data Quality of the Clinical Trial Process 179 where it ought to start: with ensuring that the variables are defined and the investigators trained on the study procedures, so that accurate measurements are made and recorded with no systematic mistakes. Instead, the coverage of GCP starts after the recordings of the measurements have been made. The notion of preventing errors is not mentioned. The idea that an investigator might make an error appears inconceivable in all applicable documents from the Declaration of Helsinki and onwards. It seems to be implied that the medical training investigators have received in their education is sufficient to prevent them from making errors and that they will retain this ability forever. Coverage of Inspections on Data Quality The standard method to keep control over the data quality in large-scale clinical trials is to inspect the data flow repeatedly, beginning with the initial recording of the data into a case report form and continuing right up until the data appear in the final study report. Large-scale studies of today encompass thousands of patients evaluated by hundreds of investigators from dozens of countries. The investigators are all supposed to perform these evaluations in a highly standardised manner. What does Quality Assurance of GCP Really Assure? The personnel responsible for clinical trial quality assurance work on the assumption that strict development and updating of the GCP documentation is a guarantee that the data quality is high. But in GCP the need for standardisation is something exceptional: in exceptional circumstances the sponsor may determine that central monitoring in conjunction with procedures such as investigators training and meetings, and extensive written guidance can assure appropriate conduct of the trial in accordance with GCP [1]. GCP does not encourage the continuous collection of data measuring the degree of adherence to the procedures set out in the GCP documentation. GCP shows more concern for the existence of data as inspected in the Source Data Verification process than with the issue of standardization of the measurement process. It is somewhat strange that this situation has developed in this scientific context. To apply GCP with incomplete coverage of the relevant quality issues is a stimulus to spend inordinate resources on doing the wrong things wasting time on symbolic checking routines that will not increase the real data quality. Thus, the scientifically meaningless formalities will be polished to perfection [3]. But, the quality management activities that matter from a scientific viewpoint are mistreated. Are there Lessons to Learn from the World Outside of Clinical Research? The Japanese industry performing mass production half a century ago started to utilize American advice to develop quality management procedures. This made them understand that their previous dependence on inspections with separate quality departments was highly inefficient. The rest of the industrialized world has learnt to appreciate, albeit with some difficulties, that the success of these methods is highly dependent on its emphasis on prevention of errors at all management levels in the production processes. Adaptation of these lessons to clinical research would mean that no investigator (in particular opinion leaders, however important) would be allowed to participate in the production of case report data without tailored and documented training for the specific study. The Role of Inspections The original (from early 1800s to about mid 1900s) approach to securing quality in mass

4 180 M. Lörstad production was based on inspections. The higher the quality requested by the customers, the more intensive became the inspections. Inspectors were organised into quality departments that were given the final responsibility for product quality. The head of the quality department reported directly to the highest level of management [4]. In the first part of last century it was realized in large parts of the industry, that this approach was too costly, as it demanded ever-growing resources. This was the initiation of statistical process control (SPC), which introduced the concept of separation of errors into random and systematic.[5] From this separation it was seen that random errors have very little effect on the final results; the focus was then changed to look out for systematic errors. After a systematic error pattern was identified, management efforts were spent on changing procedures in order to prevent systematic errors. This mode of thinking stood in stark contrast to the then traditional pattern of producing errors and then finding them. The term you cannot inspect quality into a product was coined and understood in most applications of mass production. One might venture to predict that pharmaceutical research will soon be fairly isolated in its defence of the value of inspection as a superior means of quality assurance. Total Quality Management Applying modern quality management methodology can result in efficiently attaining appropriate data quality. This methodology has been used in other logically similar applications and proven to be superior to previously used methods. Thus, evidence-based experience in other fields devaluing the value of inspections points to the need for the GCP fundamentalists to rethink. The positive effects from a good error preventive practice, so far conspicuously lacking as a concept to replace the GCP focus on inspections, would need to be supported by actual study data. Where is the research looking into these aspects? How come the pharmaceutical firms do not publish studies from such research (if they are undertaken)? In this situation, the ability of the control work to pick up and correct all the quality imperfections leaves a lot to be desired. But, if the error rates are not defined and estimated and the authorities do not ask for them, why do it? As long as documentation can show that someone has looked at the data and is able to confirm that there are no deviating data in the data flow from case report form to final table, everything is considered perfect the data quality is controlled to regulatory satisfaction. From quality management point of view, answers to the following questions are essential as a foundation for evaluation of real quality: 1. What is the definition of error? 2. How do we recognize an error when we see one? 3. Are all errors equally important? 4. Which errors cause more damage than others? 5. What rates of error are acceptable? 6. How do we continuously quantify our level of quality? None of these questions appear in quality assurance according to GCP. The Role of Variation Clinical research has a tendency to shy away from variation. Treatment efficiency is predominantly estimated in terms of arithmetic means for groups of subjects, however meaningless such an indicator might be. The focus of the reports is to show a statistical significance of the primary effect variable. Variation of effect between categories of patients is rarely quantified [6]. A basic philosophy of drug development is: Ensure homogeneity. A scaring consequence of this philosophy is that often the inference from the testing of the drug is not valid for the patient groups to whom the drug is administered. Quality management has quite a contrasting basic philosophy: Study the variation to enable identification of its sources. In early phases of product development it goes even further to stimulate the variation to be able to understand its sources.

5 Data Quality of the Clinical Trial Process 181 Quality management concentrates on analysis of variation of quality indicators of the production. The purpose of the analysis is to identify any sources of variation picked up by these indicators and to classify these sources into random and/or systematic. Random errors, by definition small and balancing each other, are accepted as a fact of life whereas sources of systematic errors give reason to change production procedures in order to eliminate them. Quality Declaration A quality declaration is a document prepared by statistics agencies for evaluation of the quality of official statistics. It does not exist in the pharmaceutical world, as yet. It gives an objective description of all processes and all steps taken to ensure that survey variables and data are defined, measured and handled in order to provide a valid, reliable and accurate database for the analyses made and conclusions drawn in the report [7]. The purpose of the quality declaration is to provide documented evidence to allow the reader and user of the data to form his/her opinion on whether the quality of the data is sufficient for the intended use of the conclusions. A quality declaration serves a scientifically based need. It stimulates necessary efforts to improve the scientific value of the information. FDA PAT Initiative In August 2002, the Food and Drug Administration (FDA) launched a methodology development initiative: the Process Analytical Technologies (PAT) initiative. It is an effort to facilitate the introduction of new technologies to the manufacturing sector of the pharmaceutical industry. The term analytical in PAT is viewed broadly to include chemical, physical, microbiological, mathematical and risk analysis conducted in an integrated manner [8]. The PAT regulatory framework gives hope for future methodological development in clinical research. The listing below is quoted from presentations made by FDA officials in international conferences on PAT. It may be seen as a reflection of the wariness of the pharmaceutical world towards any initiatives from FDA and how FDA in a preventive move tries to calm down those worries. PAT is not a requirement. Research exemption (i.e. experimental application of new methods not covered by GMP is permitted). Continuous improvements without the fear of being considered non-compliant. Regulatory support and flexibility during development and implementation. Eliminate the fear of delayed approval. Dispute avoidance/resolution. Meetings or communication with FDA. At the planning and development stage. After an initial evaluation of risk and suitability. Prior to or during the period of data generation and process understanding. Prior to submission of an application. From recent conference statements made by FDA officials it appears that discussions are being held on a number of quality systems initiatives for GCP. Conclusion Quality assurance as carried out by means of repeated inspections of data has a weak relation to the quality of data necessary for scientifically based reliability. It is a regulatory requirement intended to satisfy a Parkinsonian bureaucracy performing control activities. It is the collection of information for a document for documentation of the completeness of the regulatory documentation. If, instead, the regulatory authority would issue a requirement for a quality declaration to accompany each clinical trial, then we would have a stimulus for pharmaceutical R&D to improve on the wanting level of scientific performance and save substantial time and resources.

6 182 M. Lörstad References 1. ICH Harmonised Tripartite Guideline for Good Clinical Practice. 2. Knatterud GL, Rockhold FW, George SL, et al., Guidelines for quality assurance in multicenter trials: a position paper. Control Clin trials. 1998; 19: Lörstad, MH: From Too Much, Too Late to Right First Time ; Quality Guru Deming s Advice for Clinical Trials. Drug Inf J 2000; 34: Radford GS: The Control of Quality in Manufacturing. Ronald Press: New York, Shewhart WA: Economic Control of Quality of Manufactured Product. D van Nostrand Company: New York, Lörstad MH: Variation in Clinical Research Who Wants to Know? Conference Proceedings, First International Symposium on Industrial Statistics. Linköping: Sweden, Statistics Canada: Statistics Canada s Quality Assurance Framework, FDA, Draft Guidance for Industry A Framework for Innovative Pharmaceutical Manufacturing and Quality Assurance, August 2002.