Clinical validation of a new drug seeking Food and

THE STATISTICIAN S PAGE Prosthetic Heart Valves: Objective Performance Criteria Versus Randomized Clinical Trial Gary L. Grunkemeier, PhD, Ruyun Jin, MD, and Albert Starr, MD Providence Health System, Portland, Oregon The current Food and Drug Administration (FDA) heart valve guidance document uses an objective performance criteria (OPC) methodology to evaluate the clinical performance of prosthetic heart valves. OPC are essentially historical controls, but they have turned out to be an adequate, and perhaps optimal, study design in this situation. Heart valves have a simple open-and-close mechanism, device effectiveness is easy to document, and the common complications (thromboembolism, thrombosis, bleeding, leak, and infection) are well known and easily detected. Thus, randomized clinical trials (RCTs) have not been deemed necessary for the regulatory approval of prosthetic heart valves. The OPC are derived from the average complication rates of all approved heart valves. Studies based on OPC have been shown to work well; many different valve models have gained FDA market approval based on this methodology. Although heart valve RCTs are not required by the FDA, they have been done to compare valves or treatment regimens after approval. Recently, the Artificial Valve Endocarditis Reduction Trial (AVERT) was designed to compare a new Silzone sewing ring, designed to reduce infection, with the Standard sewing ring on a St. Jude Medical heart valve. This was the largest heart valve RCT ever proposed (4,400 valve patients, followed for as long as 4 years), but it was stopped prematurely because of a high leak rate associated with the Silzone valve. Examining the results showed that a much smaller, OPC-based study with 800 patient-years would have been sufficient to disclose this complication of the Silzone valve. (Ann Thorac Surg 2006;82:776 80) 2006 by The Society of Thoracic Surgeons Clinical validation of a new drug seeking Food and Drug Administration (FDA) approval requires a definitive randomized control trial (RCT). Prosthetic heart valve replacement devices have been in use since 1960, and many different models have been granted market approval by the FDA. Yet none of these FDA approvals required a RCT. Instead, the FDA currently relies on a sort of composite historical control, the objective performance criteria (OPC). That may seem strange, but we will argue that it is probably as it should be. See pages 773 and 1140 In doing so, we will do the following: present some general remarks about prosthetic heart valve devices; briefly describe their FDA regulatory history and the current guidelines; summarize heart valve RCTs in general and one large recent example in detail; and discuss the rationale for using OPC rather than RCT to justify their clinical approval. Herein we present the rationale for using the OPC approach as opposed to RCTs for the evaluation and approval of prosthetic heart valves. Prosthetic Heart Valves A heart valve is a one-way check valve, opening to allow the flow of blood in one direction at the appropriate time Address correspondence to Dr Jin, 9155 SW Barnes Rd, LL Suite 33, Portland, OR 97225; e-mail: ruyun.jin@providence.org. during the cardiac cycle, and sealing the valve orifice otherwise, to prevent backward flow. The first successful heart valve was the simple Starr-Edwards caged-ball device implanted in 1960 [1]. Engineering principles could be applied to the design, laboratory testing to assess flow, leak, strength, durability, and animal implantations to study the effects in a biological environment. This feedback loop resulted in rapid development and continual incremental improvements, resulting in a valve that has been used successfully for more than 40 years [2]. This type of progress would be impossible to achieve in today s regulatory environment. In the meantime, many other heart valve designs have been successfully introduced into clinical use. The most widely used to date is the St. Jude Medical (SJM) valve, with more than 1.5 million valves implanted by 2004 [3]. Clinical Results Many valve designs have been tried and abandoned at the in vitro or in vivo testing stages, or after unsatisfactory clinical usage. Those that are successful clinically have been relatively effective, as measured by diagnostic laboratory examinations: they open sufficiently to allow good forward flow and close adequately to prevent backward flow. And they are relatively safe, as measured by low rates of clinical complications. During the past 45 years, these complications, and the rates achieved by acceptable valves, have become well known. The complications have been codified in guidelines for reporting on heart valve performance, with standardized definitions simultaneously published in the United States [4, 5], 2006 by The Society of Thoracic Surgeons 0003-4975/06/$32.00 Published by Elsevier Inc doi:10.1016/j.athoracsur.2006.06.037

Ann Thorac Surg THE STATISTICIAN S PAGE GRUNKEMEIER ET AL 2006;82:776 80 PROSTHETIC HEART VALVE OPC VERSUS RCT 777 Table 1. Objective Performance Criteria (OPC) a and Patient-Year (P-Y) b Requirements Europe [6], and Asia [7]. A comprehensive literature review covering 165 reported series, using 21 different valve models and comprising 61,455 valve implants and 319,749 patient-years of follow-up demonstrated that these complications rates tend to cluster around common values [8]. Regulatory Developments Mechanical Tissue OPC P-Y OPC P-Y Thromboembolism 3.0 323 2.5 388 Valve thrombosis 0.8 1,213 0.2 4,850 All hemorrhage 3.5 277 1.4 693 Major hemorrhage 1.5 647 0.9 1,078 All paravalvular leak 1.2 808 1.2 808 Major paravalvular leak 0.6 1,617 0.6 1,617 Endocarditis 1.2 808 1.2 808 a Events per 100 valve-years (%/year). b Patient-years required to satisfy the requirement of establishing that the observed rate is less than 2 times the OPC, with power 0.80 and size 0.05. The FDA began regulating medical devices in 1976. At first, there was no standard for clinical performance, marketing applications seemed to be judged primarily on the completeness of data. Then a 1993 guidance document (revised in 1994 [9]) gave requirements for in vitro, animal, and clinical data, including New York Heart Association (NYHA) class, blood and hemodynamic data, and cardiovascular complications [10, 11]. This document included the list of OPC for valve-related complications (Table 1). The OPC are estimates of the average complication rates, based on the reports of previously approved heart valves. Current Implementation of OPC A new heart valve must be shown to have complications rates that are equivalent to the OPC. Specifically, the 1994 FDA heart valve guidance requires the sponsor to demonstrate that the observed rates for the study valve are significantly less than 2 times the OPC [9]. That is, equivalence is defined as better than twice as bad. This requirement is formulated in terms of a one-sided hypothesis test, with a size of 0.05 and a power of 0.80 [10, 11]. The OPC rates for the various complications range from 0.3% per year to 3.5% per year, and the number of patient-years thus required to fulfill the equivalency requirement range from 277 to 4,850 (Table 1). The number of patient-years needed to cover all events was deemed prohibitive, so the value of 800 patient-years was selected, to include the 9 (of 14) OPC rates equal to or greater than 1.2% per year. Many premarket approval applications (PMAs) for heart valves have been approved using the OPC approach [12]. Heart Valve RCTs Although RCTs of heart valves have not been required for FDA approval, many have been done to compare postapproval valves. We will briefly review a few of the most important ones. There are two major randomized studies of heart valves, both comparing mechanical to tissue valves. A recent RCT compared the most-used third-generation SJM bileaflet valve to the firstgeneration Starr-Edwards caged-ball valve. And the largest RCT of heart valves ever proposed, a recent comparison of a new sewing ring treatment, ended prematurely because of an increased risk, which could have been predicted by a much smaller OPC-based study. Tissue Versus Mechanical There are considered to be two major heart valve RCTs, the Edinburgh [13] and the Veterans Administration [14] studies, both comparing mechanical to tissue valves. After 12 to 15 years, they reported what had already been known for many years from numerous observational studies that tissue valves are associated with more reoperation and mechanical valves have more bleeding due to the required anticoagulant therapy. And, by the time these trials were completed, the valves used were no longer being manufactured. In fact, this was probably not the ideal comparison for a randomized study, because the valves chosen have known characteristics that make clinical equipoise unlikely. Mechanical valve failure rates are essentially zero, whereas tissue valve failure is inevitable if the patient lives long enough. But the failure rate is slower for tissue valves the older the patient, so they should be used preferentially in older patients. Mechanical valve patients have to take anticoagulant medication, which increases the risk of bleeding and imposes lifestyle limitations, whereas only a relatively small subset of tissue valves will need to do so. Thus, there are different complication profiles for the two valves, with one usually being clearly advantageous for an individual patient. Even for patients in whom the lifetime expectations of these complications is the same, most patients would still have an opinion about a constant risk of a bleeding complication versus a probable reoperation in the future. New Versus Old Among mechanical valves, the caged-ball valve is considered first generation, the single-disk valve is second generation, and the bileaflet valve is third generation. An RCT was recently completed, comparing the original silicone rubber, ball-valve technology (Starr-Edwards) to the pyrolytic carbon, bileaflet technology (SJM) [15]. These two completely different solutions to the same mechanical problem gave virtually identical clinical results out to 5 years. That two such different valvular mechanisms can be equally effective and safe supports the concept of the commonality of complications rates that is the basis of the OPC approach. In this respect, heart valves may be unique. In what other type of medical intervention would a 1960 and a 1977 technology

778 THE STATISTICIAN S PAGE GRUNKEMEIER ET AL Ann Thorac Surg PROSTHETIC HEART VALVE OPC VERSUS RCT 2006;82:776 80 be finally subjected to an RCT that is published in 2003, with the resulting conclusion of no clinical differences? Artificial Valve Endocarditis Reduction Trial The largest heart valve RCT ever proposed was the Artificial Valve Endocarditis Reduction Trial (AVERT). We will use it to demonstrate the large size requirements of an RCT, and to contrast it with the OPC approach. The SJM valve was approved by the FDA in 1982. The SJM valve with a silver-impregnated sewing ring (called the Silzone valve), designed to reduce the risk of infection, was approved in 1998 as a supplement to the original PMA. Since it was only a sewing ring modification, a full OPC-based clinical study was not required. That would seem to be the ideal use of an RCT for two similar, post market-approval valves: SJM valves with Silzone versus Standard sewing ring. The risk of endocarditis was assumed to be 0.5% per year with the Standard sewing ring valve, and the AVERT was designed to detect a 50% risk reduction with Silzone. For a size of 0.05 and power of 0.80, it required the accrual of 4,400 patients over a 3-year induction period, with an additional year of follow-up [16]. The study began in July 1998, including 17 centers in North America and Europe. Owing to an unexpected risk of explant for paravalvular leak in the Silzone arm [17], recruitment was stopped in January 2000 and the Silzone valve was discontinued, but follow-up continues to further delineate the valve s performance characteristics. Fig 1. Cumulative patient-years of follow-up (smooth, almost identical curves, indexed by the left vertical axis) and the number of major leaks (dashed step functions, indexed by the right vertical axis) for the Silzone and Standard arms of the Artificial Valve Endocarditis Reduction Trial. The scale of the left axis is 100 times that of the right axis, so that the cumulative leak curve from a valve with a rate of 1 leak per 100 patient-years (1%/year) would fall directly on top of its cumulative follow-up curve. The dotted horizontal line indicates the objective performance criteria follow-up requirement of 800 patient-years, and the unshaded area on the left indicates the follow-up observed by that time, during which there were 14 and 3 leaks in the Silzone and Standard valve arms of the study, respectively (height of the black circles, as read on the right vertical axis). Fig 2. Major leak rates with 90% confidence intervals (horizontal axis), after 800 valve-years, for various numbers of observed events (vertical axis). The dashed vertical line indicates the objective performance criteria (OPC) of 0.6%/year for major paravalvular leak. The Food and Drug Administration criterion for approval is that the upper limit of the confidence interval is less than two times the OPC (the solid vertical line). The Silzone arm (14 leaks) of the study violates the acceptance criterion, and the Standard arm (3 leaks) satisfies it (identified by the two thicker horizontal lines). Note that the upper limit of a two-sided 90% confidence interval, as plotted in this figure, is the same as the limit of a one-sided 95% confidence interval, as required by the FDA guidance document. The AVERT study, when finished, would have included more than 10,000 patient-years. Ironically, a much smaller OPC-based study, with 800 patient-years, would have been sufficient to identify the increased Silzone risk of major leak. The cumulative follow-up years and number of major leaks for each arm of the study through July 2005 is shown in Figure 1. By the time 800 patient-years had accrued in each arm of the study, there were 3 major leaks in the Standard arm and 14 major leaks in the Silzone arm. Figure 2 shows the leak rates and their 90% confidence intervals, on the horizontal axis, produced by varying numbers of major leaks in 800 patient-years, on the vertical axis. Figure 2 shows that the leak rate with the Silzone valve is significantly higher than the OPC (fails the OPC test). Figure 2 also shows that the Standard valve passes the OPC test, as its leak rate is significantly less than twice the OPC. Fortunately, the excess risk with Silzone has disappeared over time, and the risk by about 2 years after implant is equal to that of the Standard valve [18]. In actuality, the AVERT trial was stopped in January 2000, long before 800 patient-years were accrued with the Silzone valve (there was a potential of about 300 patientyears by that time, as can be seen from Fig 1). By that time, there were 9 major leaks reported for Silzone and 3 for the control. Thus, it can be seen from Figure 2 that, even if 800 patient-years had been accumulated without any more major leaks, the Silzone valve still would not have passed the OPC test (by strict interpretation, only 4 events are permitted). That raises the issue of a sequential, rather than fixed sample size study, which could well be more appropriate for medical devices.

Ann Thorac Surg THE STATISTICIAN S PAGE GRUNKEMEIER ET AL 2006;82:776 80 PROSTHETIC HEART VALVE OPC VERSUS RCT 779 Rationale for OPC: Drugs Versus Devices There are major differences between drugs and heart valves, which may explain why RCTs are required for regulatory approval of the former, but not the latter. The evaluation stages of drug studies are (1) establish safety, (2) refine dosage, and (3) confirm efficacy. The last step is difficult because usually the mechanism of action is systemic, complex, and not well understood, and the endpoints are partly subjective. Consequently, many patients are needed, with randomization and blinding necessary to avoid bias. Also, drug treatment is temporary and reversible; the patient can discontinue the inferior drug and crossover to the other treatment. Indeed, crossover is often a planned element in a drug study design, but cannot be considered in a heart valve study. With many devices, and heart valves in particular, determining effectiveness is less problematic. The mechanism is simple, local, and well understood. The evaluation stages are establish efficacy (there is no dosage), and confirm safety. The endpoints are objective, there is no placebo effect, study patients are relatively few, and the treatment is permanent. Masking/blinding cannot be used, and there are problems with establishing a control. Some authors have argued that observational studies are not necessarily inferior to RCTs [19] and that careful observational studies can yield the same conclusions as RCTs [20]. Arguments can be made in favor of observational studies for heart valves [21], and some clinicians agree that RCTs are not ideal in this setting [22, 23]. Others have concluded that the results from RCTs do not generally reflect real-life clinical situations [24]. Over the years, prospective observational studies have preformed well in the evaluation of artificial heart valves. The Silzone valve was not subjected to a clinical study for regulatory approval but, by the analysis presented above, if one had been done, it would have failed the OPC test for major paravalvular leak. The OPC method depends on identifying all valverelated complications, including those that cause death, so that a high autopsy percentage must be a pivotal requirement of the study. That may not be as important in an RCT, since total mortality and sudden unexplained deaths can be compared between the study and control groups. Because there are no OPC for these events, it is important that fatal valve-related events be classified into the OPC category to which they belong. A requirement for using simple OPC rates (percent per year) to summarize a complication is that its risk is constant across time. When this assumption is true, it does not matter how long the follow-up times are. A 5-year follow-up of 160 patients and a 6-month follow-up of 1,600 patients, for example, both yield 800 patient-years. But if a complication is time-related, then not only does the constant risk assumption not hold, but the length of follow-up can be critical. An example is structural failure of bioprosthetic heart valves. That is not an OPC complication because it has an increasing risk, and to provide useful estimates of it would take follow-up times (12 to 15 years?) that are impractical given the development cycle of new valve technology. For these reasons, the OPC approach may not be appropriate for all medical devices. Conclusions and Future Developments The OPC approach has been shown to work well for regulatory approval of heart valves. Randomized clinical trials are not required for establishing their acceptability for clinical use. However, more sophisticated single-arm observational studies could be developed, possibly using Bayesian methods (Appendix), as previously suggested for heart valves [25] and coronary artery stents [26]. The FDA has considered and approved medical devices based on studies using Bayesian methods [27, 28], and just released a Draft Guidance for the Use of Bayesian Statistics in Medical Device Clinical Trials [29]. The next revision of the clinical requirement of the FDA guidance document for heart valves might incorporate such techniques. The authors are grateful to the reviewers of this editorial for insightful suggestions that improved its presentation and content. Appendix Bayesian Statistics The basic difference between the Bayesian and the more commonly used frequentist statistical approaches exists at the interface between philosophy and science [12]. Bayesian methods are older, have better theoretical properties, and are more intuitive. Frequentist methods rely only on the data produced in a given experiment to draw conclusion, while Bayesian methods formally integrate previously available information into the process (as we all do when making a decision). A simple example: suppose you have a jar full of red and black jellybeans and want to estimate the percentage of red, based on the first 10 jellybeans you draw of which 7 are red and 3 are black. In the absence of any other information, your best estimate of the percentage of red jellybeans in the jar would be 70%. Now suppose you have a coin and want to guess the true percentage of heads based on the first 10 tosses of which 7 are heads and 3 are tails. The data are the same, but do you really believe the chance of heads is 70%? No, because your prior experience with coins tells you that the true probability is very unlikely to be this extreme. A Bayesian would incorporate past experience with coins into her estimate to arrive at a percentage somewhere between 50% and 70%. This intuitive property has caused Bayesian methods to be used for many years in the interpretation of diagnostic tests [30]. Another advantage that Bayesian statisticians enjoy is a completely flexible stopping rule in a clinical trial, so that the AVERT Data Safety and Monitoring Board would have had a firm basis for stopping the trial prematurely, rather than the agonizing process they no doubt went through. And Bayesians are allowed to make direct probability statements about the results, such as the probability that the true value is between X and Y is 95%. These are not possible with frequentist methods, though most investigators erroneously interpret confidence intervals this way. Wide use of Bayesian techniques has been held back by the

780 THE STATISTICIAN S PAGE GRUNKEMEIER ET AL Ann Thorac Surg PROSTHETIC HEART VALVE OPC VERSUS RCT 2006;82:776 80 lack of widely available software; they are not yet found in the most common statistical packages, but that limitation is beginning to be overcome. References 1. Starr A, Edwards ML. Mitral replacement: clinical experience with a ball-valve prosthesis. Ann Surg 1961;154:726 40. 2. Gao G, Wu Y, Grunkemeier GL, Furnary AP, Starr A. Forty-year survival with the Starr-Edwards heart valve prosthesis. J Heart Valve Dis 2004;13:91 6. 3. Concato J. Prostate specific antigen: a useful screening test? Cancer J 2000;6(Suppl):188 92. 4. Edmunds LH Jr, Clark RE, Cohn LH, Grunkemeier GL, and mortality after cardiac valvular operations. Ad Hoc Liaison Committee for standardizing definitions of prosthetic heart valve morbidity of the American Association for Thoracic Surgery and The Society of Thoracic Surgeons. J Thorac Cardiovasc Surg 1996;112:708 11. 5. Edmunds LH Jr, Clark RE, Cohn LH, Grunkemeier GL, and mortality after cardiac valvular operations. The American Association for Thoracic Surgery, Ad Hoc Liason Committee for standardizing definitions of prosthetic heart valve morbidity. Ann Thorac Surg 1996;62:932 5. 6. Edmunds LH Jr, Clark RE, Cohn LH, Grunkemeier GL, and mortality after cardiac valvular operations. Eur J Cardiothorac Surg 1996;10:812 6. 7. Edmunds LH Jr, Clark RE, Cohn LH, Grunkemeier GL, and mortality after cardiac valvular operations. Asian Cardiovasc Thorac Ann 1996;4:126 9. 8. Grunkemeier GL, Li HH, Naftel DC, Starr A, Rahimtoola SH. Long-term performance of heart valve prostheses. Curr Probl Cardiol 2000;25:73 154. 9. Division of Cardiovascular and Neurological Devices. Draft Replacement Heart Valve Guidance, Version 4.1. Washington, DC: Center for Devices and Radiological Health, Food and Drug Administration; 1994:1 134. 10. Johnson DM, Sapirstein W. FDA s requirements for in-vivo performance data for prosthetic heart valves. J Heart Valve Dis 1994;3:350 5. 11. Sapirstein W. Designing trials for testing prosthetic cardiac valves: a Food and Drug Administration perspective. Am Heart J 2001;141:861 3. 12. Bland JM, Altman DG. Bayesians and frequentists. Br Med J 1998;317:1151 60. 13. Oxenham H, Bloomfield P, Wheatley DJ, et al. Twenty year comparison of a Bjork-Shiley mechanical heart valve with porcine bioprostheses. Heart 2003;89:715 21. 14. Hammermeister K, Sethi GK, Henderson WG, Grover FL, Oprian C, Rahimtoola SH. Outcomes 15 years after valve replacement with a mechanical versus a bioprosthetic valve: final report of the Veterans Affairs randomized trial. J Am Coll Cardiol 2000;36:1152 8. 15. Murday AJ, Hochstitzky A, Mansfield J, et al. A prospective controlled trial of St. Jude versus Starr Edwards aortic and mitral valve prostheses. Ann Thorac Surg 2003;76:66 74. 16. Schaff H, Carrel T, Steckelberg JM, Grunkemeier GL, Holubkov R. Artificial Valve Endocarditis Reduction Trial (AVERT): protocol of a multicenter randomized trial. J Heart Valve Dis 1999;8:131 9. 17. Schaff HV, Carrel TP, Jamieson WR, et al. Paravalvular leak and other events in Silzone-coated mechanical heart valves: a report from AVERT. Ann Thorac Surg 2002;73:785 92. 18. Grunkemeier GL, Jin R, Im K, Holubkov R, Kennard ED, Schaff HV. Time-related risk of the St. Jude Silzone heart valve. Eur J Cardiothorac Surg 2006;30:20 7. 19. Concato J. Observational versus experimental studies: what s the evidence for a hierarchy? NeuroRx 2004;1:341 7. 20. Benson K, Hartz AJ. A comparison of observational studies and randomized, controlled trials. N Engl J Med 2000;342: 1878 86. 21. Grunkemeier GL, Starr A. Alternatives to randomization in surgical studies. J Heart Valve Dis 1992;1:142 51. 22. Bach DS. Choice of prosthetic heart valves: update for the next generation. J Am Coll Cardiol 2003;42:1717 9. 23. Wheatley DJ. Clinical evaluation: statistical considerations and how to meet them in clinical practice. J Heart Valve Dis 2004;13(Suppl 1):11 3. 24. Grapow MT, von Wattenwyl R, Guller U, Beyersdorf F, Zerkowski HR. Randomized controlled trials do not reflect reality: real-world analyses are critical for treatment guidelines. J Thorac Cardiovasc Surg 2006;132:5 7. 25. Grunkemeier GL, Payne N. Bayesian analysis: a new statistical paradigm for new technology. Ann Thorac Surg 2002; 74:1901 8. 26. O Malley AJ, Normand SL, Kuntz RE. Application of models for multivariate mixed outcomes to medical device trials: coronary artery stenting. Stat Med 2003;22:313 36. 27. Temple R. How FDA currently makes decisions on clinical studies. Clin Trials 2005;2:276 81. 28. Campbell G. The experience in the FDA s Center for Devices and Radiological Health with Bayesian strategies. Clin Trials 2005;2:359 63. 29. Guidance for the Use of Bayesian Statistics in Medical Device Clinical Trials Draft Guidance for Industry and FDA Staff, 2006. Available at: http://www.fda.gov/cdrh/osb/ guidance/1601.html (accessed Aug 2, 2006). 30. Gill CJ, Sabin L, Schmid CH. Why clinicians are natural Bayesians. Br Med J 2005;330:1080 3.