Developing and Validating an In Silico Model for Proarrhythmia Risk Assessment Under the CiPA Initiative

Transcription

1 Developing and Validating an In Silico Model for Proarrhythmia Risk Assessment Under the CiPA Initiative May 2018 Zhihua Li, PhD for the CiPA In Silico Working Group US Food and Drug Administration This presentation reflects the views of the author and should not be construed to represent FDA s views or policies

2 Comprehensive in vitro Proarrhythmia Assay (CiPA) I Ks and I Na Peak in specific situations Torsade Metric Score Check for unanticipated human effects, confirm mixed channel effects Can be considered for unanticipated nonclinical effects, or if human ECG data is insufficient

3 Model Development and Validation Strategy Select a Base Cardiomyocyte Model CiPA Training Drugs (12) Model Optimization Model Training Metric Development Evaluate the Training Results; Freeze Model for Validation CiPA Validation Drugs (16) Predict Validation Drugs Compare Prediction Accuracy to Predefined Performance Measures Model Validation

4 Model Development and Validation Strategy Select a Base Cardiomyocyte Model CiPA Training Drugs (12) Model Optimization Metric Development Model Training Evaluate the Training Results; Freeze Model for Validation Predict Validation Drugs CiPA Validation Drugs (16) Compare Prediction Accuracy to Predefined Performance Measures Model Validation 4

5 Improving the ORd Model for CiPA Modeling dynamic drug-herg interactions rather than using simple IC50s to distinguish drugs with similar herg block potency but different TdP liabilities Li Z et al. Circulation: Arrhythmia & Electrophysiology. 2017;10:e Optimizing model parameters so that the model can better recapitulate experimental data on human ventricular myocytes Dutta et al. Frontiers in Physiology. 2017;8:616 5

6 Model Development and Validation Strategy Select a Base Cardiomyocyte Model CiPA Training Drugs (12) Model Optimization Model Training Metric Development Evaluate the Training Results; Freeze Model for Validation CiPA Validation Drugs (16) Predict Validation Drugs Compare Prediction Accuracy to Predefined Performance Measures Model Validation 6

7 Key Mechanism of TdP: imbalance of Inward and Outward Currents Early after depolarization (EAD) plateau Action potential Torsade de pointes ECG Increased ratio between inward and outward currents QT Major currents modulating repolarization Inward Outward ICaL (L type calcium) IKr (potassium) The net current between inward and outward currents reflect their balance. INaL (late sodium) IKs (potassium) IK1 (potassium) Ito (potassium) Inet = ICaL+INaL+IKr+IKs+IK1+Ito qnet: Amount of electronic charge carried by Inet 7

8 Performance of qnet on 12 CiPA Training Compounds Arrhythmia Metric (qnet) EAD generated Low Risk Intermediate Risk High Risk Drug concentration (fold change over clinical exposure) Simulation with 2000 ms cycle length Drug separation is good along all concentrations from 1x to 25x Cmax 8

9 Uncertainty Quantification for TdP Risk Assessment under CiPA Developed a statistical method to translate each drug s experimental uncertainty into 2000 metric values, describing the probability distribution of its TdP risk Found that uncertainty is lowest when drug concentration is 1-4x Cmax 9

10 Torsade Metric Score for Manual Training Data 95%CI and median point of each drug s 2000 scores are shown as error bars High risk Intermediate risk Low risk Torsade Metric Score (qnet averaged 1-4 Cmax) herg (potassium channel) data: manual patch clamp Non-hERG (sodium and calcium channel) data: manual patch clamp 10

11 Torsade Metric Score for Hybrid Training Data 95%CI and median point of each drug s 2000 scores are shown as error bars High risk Intermediate risk Low risk Torsade Metric Score (qnet averaged 1-4 Cmax) herg (potassium channel) data: manual patch clamp Non-hERG (sodium and calcium channel) data: automated high throughput patch clamp systems 11

12 Model Development and Validation Strategy Select a Base Cardiomyocyte Model CiPA Training Drugs (12) Model Optimization Model Training Metric Development Evaluate the Training Results; Freeze Model for Validation CiPA Validation Drugs (16) Predict Validation Drugs Compare Prediction Accuracy to Predefined Performance Measures Model Validation 12

13 Evaluating and Freezing Model Prior to Validation On March 15 th 2017, FDA held a Pharmaceutical Science and Clinical Pharmacology Advisory Committee Meeting on the topic of Model Informed Drug Development, where CiPA was presented as a potential new regulatory paradigm to seek external expert opinions A Validation Procedure document was vetted by CiPA In Silico Working Group and Ion Channel Working Group, and approved by Steering Committee prior to validation The published CiPAORdv1.0 model and qnet (Torsade Metric Score) metric, as well as classification thresholds, were frozen Defined two validation datasets: one manual and one hybrid, each 16 drugs Defined two types of performance measurements: ranking TdP risk without specific classification thresholds, and classifying drugs into one of the three risk categories using specific thresholds 13

14 Model Development and Validation Strategy Select a Base Cardiomyocyte Model CiPA Training Drugs (12) Model Optimization Model Training Metric Development Evaluate the Training Results; Freeze Model for Validation CiPA Validation Drugs (16) Predict Validation Drugs Compare Prediction Accuracy to Predefined Performance Measures Model Validation 14

15 Overall Performance on Validation Drugs High Intermediate Low Torsade Metric Score (qnet averaged 1-4 Cmax) 15

16 Rank Performance: AUC of ROC ROC (Receiver Operating Characteristic) : a curve of sensitivity vs 1-specificity for all possible cut-off (thresholds) of the metric Area Under the Curve (AUC) of ROC: probability of ranking a higher risk drug above a lower risk one ROC1: Low risk vs High-or-Intermediate ROC2: High vs Low-or-Intermediate 16

17 Procedure of ROC1 Analysis 16 validation drugs, each with 2000 torsade metric scores Two categories: High-or-Intermediate Risk; Low Risk Randomly take one score per drug Rank the 16 scores (red: High-or-Intermediate; blue: Low Risk) Lowest Score Highest Score Repeat 10,000 times Construct ROC curve Calculate AUC 17

18 Rank Performance using ROC1: Low vs High-or-Intermediate One of ROC Curves Analyze 10,000 ROC curves AUC (95% CI): 0.98 (0.93 1) AUC = 0.98 Excellent performance Performance Interpretation Minimally Good Excellent Measure acceptable AUC of ROC1 >~0.7 >~0.8 >~0.9 Probability of ranking an Intermediateor-High risk drug above a Low risk drug 18

19 Rank Performance using ROC2: High vs Low-or-Intermediate One of ROC Curves AUC = 0.96 Analyze 10,000 ROC curves AUC (95% CI): 0.94 ( ) Excellent performance Performance Interpretation Minimally Good Excellent Measure acceptable AUC of ROC2 Probability of ranking a High risk drug >~0.7 >~0.8 >~0.9 above an Intermediate-or- Low risk drug 19

20 Rank Performance using Pairwise Comparison This measure does not reduce three risk categories into two (so more comprehensive) 28 CiPA drugs -> 378 possible pairwise comparisons Removing within-category drug pairs Removing training-only drug pairs Resulting in 211 drug pairs for validation Use the model to predict pairwise ranking for each drug pair, comparing results to known ranking of TdP risk Correct prediction fraction among the 211 pairs indicates ranking performance across all 3 categories Repeat 10,000 times through random sampling, estimate confidence interval of the correct prediction fraction 20

21 Rank Performance using Pairwise Comparison One of the 10,000 prediction of 211 Drug Pairs Green: drug pairs whose ranking predicted correctly Red: drug pairs whose ranking predicted incorrectly Correct prediction fraction in this analysis: 0.97 After 10,000 repeats: 0.96 ( ) Excellent performance Performance Measure Pairwise comparison Interpretation Probability of correctly ranking a drug relative to CiPA reference drugs across 3 categories Minimally acceptable Good Excellent >~0.7 >~0.8 >~0.9 21

22 Ranking Performance Performance Measure Interpretation Manual Dataset Hybrid Dataset AUC of ROC1 Probability of ranking an Intermediate-or ( ) 0.98 (0.93 1) High risk drug above a Low risk drug AUC of ROC2 Probability of ranking a High risk drug above an Intermediate-or-Low drug 1 (0.92-1) 0.94 ( ) Pairwise Ranking Probability of correctly ranking a drug relative to CiPA reference drugs through 0.95 ( ) 0.96 ( ) pairwise comparison Below minimally acceptable Minimally acceptable Good Excellent For both manual and hybrid datasets, ranking performance of Torsade Metric Score all reached or are very close to excellent level. 22

23 Classification Performance: Likelihood Ratio (LR) Likelihood Ratio positive (LR+): how much more likely a higher risk drug is classified into the higher risk category compared to a lower risk drug Likelihood Ratio negative (LR-): 1/LR- indicates how much less likely a higher risk drug is classified into the lower risk category compared to a lower risk drug Likelihood Ratio was calculated for each of the two thresholds pre-determined by training data : threshold 1 separating low from the rest while threshold 2 separating high from the rest 23

24 Classification Performance : Likelihood Ratio of Threshold 1 95%CI and median point of each drug s 2000 scores are shown as error bars High-or-Intermediate Low Risk The hybrid validation dataset result is shown here as an example Torsade Metric Score (qnet averaged 1-4 Cmax) For LR analysis of Threshold 1, High and Intermediate combined as one category (red) 10,000 LR analyses are repeated by sampling Torsade Metric Score distributions 24

25 Classification Performance : Likelihood Ratio of Threshold 1 LR+ of Threshold 1 for Hybrid Dataset: 8e5 (7e5 1e6) 1/LR- of Threshold 1 for Hybrid Dataset: 5.5 (3.7-1e6) Performance Interpretation Minimally Good Excellent Measure acceptable LR+ of Threshold 1 How much more likely a High-or-Intermediate drug will be predicted as High-or-Intermediate, compared to a Low Risk drug? >~2 >~5 >~10 1/LR- of Threshold 1 How much less likely a High-or-Intermediate drug >~2 >~5 >~10 will be predicted as Low Risk, compared to a Low Risk drug? Wide confidence interval (CI) of 1/LR- due to perfect classification of some samples, leading to upper 95%CI value close to infinity. 25

26 Classification Performance : Likelihood Ratio of Threshold 2 LR+ of Threshold 2 for Hybrid Dataset : 6 (3 12) 1/LR- of Threshold 2 for Hybrid Dataset: 3.7 (3. 9e5) Performance Interpretation Minimally Good Excellent Measure acceptable LR+ of Threshold 1 How much more likely a High Risk drug will be predicted as High Risk, compared to a Low-or- Intermediate Risk drug? >~2 >~5 >~10 1/LR- of Threshold 1 How much less likely a High Risk drug will be >~2 >~5 >~10 predicted as Low-or-Intermediate, compared to a Low or-intermediate Risk drug? Wide confidence interval (CI) of 1/LR- due to perfect classification of some samples, leading to upper 95%CI value close to infinity. 26

27 Mean Classification Error High Intermediate Low Mean error across 16 drugs x 2000 samples 0.25 ( ) Torsade Metric Score (qnet averaged 1-4 Cmax) Performance Interpretation Minimally Good Excellent Measure Mean Classification Error Average error of classifying each of the 16 validation drugs into High, Intermediate, or Low risk category acceptable <~1 <~0.5 <~0.3 27

28 Classification Performance Performance Measure Interpretation Manual Dataset HybridDataset LR+ of Threshold 1 1/LR- of Threshold 1 LR+ of Threshold 2 1/LR- of Threshold 2 How much more likely a High-or-Intermediate drug will be predicted as High-or-Intermediate, compared to a Low Risk drug? How much less likely a High-or-Intermediate drug will be predicted as Low Risk, compared to a Low Risk drug? How much more likely a High Risk drug will be predicted as High Risk, compared to a Low-or-Intermediate Risk drug? How much less likely a High Risk drug will be predicted as High Risk, compared to a Low or-intermediate Risk drug? 4.5 (2.3 5) 8e5 (7e5 1e6) 8.8 (4.4 8e5) 5.5 (3.7 1e6) 12 (4.5 1e6) 6 (3 12) 9e5 (3.3 1e6) 3.7 (3 9e5) Mean Classification Error Average error of classifying each of the 16 validation drugs into 0.19 ( ) 0.25 ( ) High, Intermediate, or Low risk category Below minimally acceptable Minimally acceptable Good Excellent For classification measures, Torsade Metric Score on the manual and hybrid datasets mostly hit good to excellent performance. 28

29 Comparing to Alternative Metrics: Ranking Performance Performance measure Dataset Torsade Metric Score APD90 APD50 & diastolic Ca AUC of ROC1 Manual ( ) ( ) ( ) Hybrid ( ) ( ) ( ) AUC of ROC2 Manual (0.95-1) ( ) ( ) Hybrid ( ) ( ) ( ) Pairwise Comparison Correct Rate Manual ( ) Hybrid ( ) ( ) ( ) ( ) ( ) Below minimally acceptable Minimally acceptable Good Excellent All 28 drugs and leave-one-out cross validation were used. CiPA metric (qnet/tms) has excellent (>0.9) performance over all datasets and measures. 29

30 Comparing to Alternative Metrics: Classification Performance Performance measure LR+ of Threshold 1 1/LR- of Threshold 1 LR+ of Threshold 2 LR- of Threshold 2 Mean Classification Error Dataset Torsade Metric Score Manual 8.05 (4.03-9) Hybrid 8.05 ( e+05) Manual 14.7 (5.6 8e5) Hybrid 14.7 (6.3 8e5) Manual 7.5e+05 (8.75-1e+06) Hybrid 15 ( ) Manual 4 (3.8 1e6) Hybrid 3.8 ( ) Manual ( ) Hybrid ( ) APD ( ) 2.68 ( ) 5.3 ( ) 6.3 ( ) 15 ( ) 15 ( ) 3.8 ( ) 3.8 ( ) ( ) ( ) APD50 & diastolic Ca 4.03 ( ) 3.55 ( ) 7.4 ( ) 4.9 ( ) 17.5 ( e+05) 15 ( ) 4 (3.8 9e5) 3.8 ( ) ( ) ( ) Below minimally acceptable Minimally acceptable Good Excellent 30

31 Summary The CiPA model adopts the most stringent validation strategy for evaluating TdP risk prediction accuracy Over two validation datasets, the CiPA model/metric generally reaches pre-defined excellent ranking performance (5 times excellent and 1 time good), and generally good to excellent classification performance (5 times excellent, 3 good, and 2 minimally acceptable). The model s ability to achieve high performance levels across both datasets despite the data differences shows its flexibility and robustness in handling heterogeneous sources of data The current CiPA model/metric outperforms all alternative metrics tested Taken together, this work supports the regulatory use of the CiPA model 31

32 Updated CiPA Drugs Torsade Metric Scores and Thresholds 95%CI of each drug s 2000 scores are shown as error bars High Intermediate Low Torsade Metric Score (qnet averaged 1-4 Cmax) All 28 drugs were used to update the classification thresholds Some drugs in vitro data were updated with higher quality data to better reflect the underlying pharmacology 32

33 Acknowledgements CiPA Steering Committee Ayako Takei, Bernard Fermini, Colette Strnadova, David Strauss, Derek Leishman, Gary Gintant, Jean-Pierre Valentin, Jennifer Pierson, Kaori Shinagawa, Krishna Prasad, Kyle Kolaja, Natalia Trayanova, Norman Stockbridge, Philip Sager, Tom Colatsky, Yuko Sekino, Zhihua Li, Gary Mirams All CiPA Working groups Ion Channel working group In silico working group Cardiomyocyte working group Phase 1 ECG working group ALL contributors to CiPA (there are a lot!) Public-private partnerships: HESI, SPS, CSRC Regulatory Agencies: FDA, EMA, PMDA/NIHS, Health Canada Many pharmaceutical, CRO, and laboratory device companies Academic collaborators FDA Contributors Norman Stockbridge Christine Garnett David Strauss Zhihua Li Wendy Wu Sara Dutta Phu Tran Jiangsong Sheng Kelly Chang Kylie Beattie Xiaomei Han Bradley Ridder Min Wu Aaron Randolph Richard Gray Jose Vicente Lars Johannesen

34 Back-Up 34

35 Improving the ORd Model for CiPA Making the IKr/hERG component temperature dependent Modeling dynamic drug-herg interactions rather than using simple IC50s Optimizing model parameters based on experimentally recorded drug effects on human ventricular myocytes 35

36 Modeling Dynamic drug-herg Interactions Rather Than IC50s Drug block of ion channels is dynamic and can exhibit a marked dependence on voltage, time, pulse frequency and channel state (open, closed, inactivated) Measurements of IC50s are therefore only snapshots of the drug-channel interaction Drugs with similar IC50 values but different kinetics for channel blocking (and unblocking) may carry different levels of risk for producing TdP (dofetilide vs. cisapride) Specially designed voltage protocol and model are needed to capture dynamic drug-channel interaction 36

37 Dynamic herg Model to Capture Drug Binding Kinetics O = Open C = Closed I = Inactivated Li Z et al. Improving the In Silico Assessment of Proarrhythmia Risk by Combining herg-drug Binding Kinetics and Multichannel Pharmacology. In preparation Model can capture dynamic drug-herg interactions, especially drug being trapped within closed channel (red arrows) Different drugs have different propensity to be trapped in the closedbound state, leading to different TdP liability 37

38 Dynamic herg Protocol Recommended by Ion Channel Working Group Voltage protocol close -80 mv open 0 mv close -80 mv Modified from Milnes et al. J. Pharmacol. Toxicol. Methods herg Current (I) Fractional Block 1 II dddddddd II cccccccccccccc Fractional Block Time (s) 38

39 Block Development for Drugs with Different Binding Kinetics close -80 mv open 0 mv close -80 mv open 0 mv close -80 mv open 0 mv No block recovery during channel closing No block recovery during channel closing Dofetilide (highly trapped): Block recovery during channel closing Block recovery during channel closing Cisapride (less trapped): 39

40 Improving the ORd Model for CiPA Making the IKr/hERG component temperature dependent Modeling dynamic drug-herg interactions rather than using simple IC50s Optimizing model parameters based on experimentally recorded drug effects on human ventricular myocytes 40

41 Optimizing ORd Model Parameters Based on Human Cardiomyocyte Data Action Potential Duration (ms). Experimental data Before optimization After optimization Goal: adjust model parameters to more faithfully recapitulate experimental data Human cardiomyocyte action potential duration (APD) was recorded under L- type calcium current (ICaL) blocker (1 µm nisoldipine) The optimized model was able to reproduce the experimental data better than the original model Cycle Length (ms) Dutta S et al. Optimization of an In Silico Cardiac Cell Model for Proarrhythmia Risk Experimental data were taken from O Hara et al. PloS Computational Biology

42 qnet vs APD : A Case Study Q: Which cell is in a more dangerous status (closer to EAD generation)? APD: The cell with ranolazine (black) qnet: The cell with cisapride (grey) Applying the same pro-ead push Added 91.6% IKr conductance reduction (perturbation) Time (ms) Time (ms) qnet, but not APD, correctly predicts the distance from EAD qnet, but not APD, independently supports the rank order of the two drugs in CiPA categories 42

43 Comparing to Other Metrics New metrics APD Based metrics qnet Training Error Other metrics Drug Conc. (fold change over Cmax) qnet is the only metric with 0 training error across all concentrations Metrics based on action potential duration (APD), the cellular basis for QT interval, failed to classify all training drugs 43

44 ncorporating Experimental Uncertainty Experimental data have intrinsic (i.e. inherent randomness) and extrinsic (i.e. cell-to-cell variability) uncertainty This will lead to uncertainty in metric calculation and TdP risk assessment A method is needed to incorporate experimental uncertainty and calculate a probabilistic distribution of the metric 44

45 Find the Optimal Concentrations for CiPA Prediction Prediction error based on leave-one-out cross validation At each concentration, there are 12 errors, corresponding to 12 training drugs Black line: mean error across 12 training drugs Lowest prediction error achieved for 1-4x Cmax Conclusion: For CiPA manual training dataset, concentrations 1-4x Cmax should be used for qnet calculation and TdP risk prediction. 4545

46 Relationship Between In Vitro Data and In Silico Model Prediction The two datasets (Manual and Hybrid) were generated using very different experimental protocols and quality control procedures, leading to some in vitro data differences between them Internal investigation suggests dataset specific bias might be the cause of some mismatches between model prediction and actual CiPA risk Standardizing protocols and quality control criteria (ongoing effort) can potentially further increase model prediction accuracy. The model s ability to achieve high performance levels across both datasets despite the data differences shows its flexibility and robustness in handling heterogeneous sources of data

47 A Possible Explanation of the Outliers Manual Dataset Hybrid Dataset INaL IC50 (µm) INaL IC50 (µm) IC50 underestimated? Disopyramide Risk underestimated in Hybrid Metoprolol IC50 overestimated? Risk overestimated in Manual Systematic difference: 25 out of 28 drugs have a higher INaL IC50 in Manual than in Hybrid Dataset Manual Dataset Hybrid Dataset ICaL IC50 (nm) ICaL IC50 (nm) Domperidone IC50 overestimated? Risk overestimated in Hybrid Systematic difference: 21 out of 28 drugs have a higher ICaL IC50 in Hybrid than in Hybrid Dataset Outliers are dataset specific, and associated with systematic in vitro data bias due to different experimental conditions and quality criteria Establishing standard experimental procedures and quality criteria may further increase prediction accuracy 47

48 Classification Performance : Likelihood Ratio of Threshold 2 95%CI and median point of each drug s 2000 scores are shown as error bars High Risk Intermediate-or-Low The hybrid validation dataset result is shown here as an example Torsade Metric Score (qnet averaged 1-4 Cmax) For LR analysis of Threshold 2, Low and Intermediate combined as one category (blue) 10,000 LR analyses are repeated by sampling Torsade Metric Score distributions 48

49 Manual herg dynamic + Manual non-herg IC50s High Intermediate Low Torsade Metric Score (qnet averaged 1-4 Cmax) 49

50 HTS herg IC50 + HTS non-herg IC50s High Intermediate Low Torsade Metric Score (qnet averaged 1-4 Cmax) 50

51 Manual herg IC50 + Manual non-herg IC50s High Intermediate Low Torsade Metric Score (qnet averaged 1-4 Cmax) 51