To: cc: From: VQA Advisory Board (VQAAB) Members Mike Ussery Joe Fitzgibbon VQA Members Cheryl Jennings Date: 30 July 2010 Subject: VQAAB Conference Call Minutes (call dated 13 July 2010) Attendees: Voting: Bill Meyer (Chair), Walter Scott (IMPAACT), Danielle Jones (HPTN),Susan Fiscus (IMPAACT), Urvi Parikh (MTN), Joan Dragavon (ACTG/HVTN), Belinda Yen-Lieberman (ACTG), Robert Coombs (ACTG/HVTN), Non-voting: Errance Meeks (IMPAACT), Rachana Kshatriya (IMPAACT), Kristi Kaiss (SSS), Ronald Bosch (SDAC), James Bremer (VQA), Heather Sprenger (FSTRF), Brian Harty (NERI), Suzanne Granger (NERI), Paige Etter (HANC), Joe Fitzgibbon (DAIDS), Cheryl Jennings (VQA), Akbar Shahkolahi (SSS), Carrie Wager (NERI) Regrets: Mike Ussery (DAIDS), Don Brambilla (RTI), Meena Doshi (NERI), Agenda: 1. Roll Call 2. Review of June 2010 Conference Call Minutes (Cheryl Jennings) 3. "Canned comments" for HIV reports (Joan Dragavon) 4. Memo to Labs: Specimen Dilution (Cheryl Jennings) 5. Memo to Labs: Manual Abbott Validation Plan (Cheryl Jennings) 6. Proposed changes for HIV RNA PT scoring during the transition from citrated plasma matrix to a SERUM+EDTA matrix (Carrie Wager) 7. Review of New Proficiency Panel Reports (Suzanne Granger) 8. Update on the external HIV RNA 200 copies/ml control results (Suzanne Granger) 9. Change in Laboratory Status Report (Cheryl Jennings) 10. New Items of Interest from the VQA (Jim Bremer/Cheryl Jennings) 11. New Items of Interest from the Committee Members Materials: VQAAB conference call minutes 06-21-2010b VQAAB RNA samples 21 1-22 10_second report VQAAB_DNA185_first report Cumulative Ratings blinded 07-06-2010 [Proposed changes for HIV RNA PT scoring] (Carrie) Abbott RNA report canned text scenarios-2 VQA_Specimen_Dilution_Plan_02July2010 VQA_Manual_Abbott_Validation_Plan_08July2010 Discussion: Created on 8/13/2010 1:07:00 PM 1 of 5
1) Roll Call 2) Review of Conference Call Minutes. The minutes from the June call were submitted to the committee for review. They will be given one week to comment before approved for posting. 3) Canned Comments for HIV Reports. Joan Dragavon presented a third version of the canned text that would be utilized in the LDMS to address different testing scenarios. The purpose of the document is to standardize the LDMS reports to contain all the relevant information/explanations that need to accompany the types of results obtained with the new HIV RNA platforms. This includes the language that will be used for detectable samples with results below the lower detection limit, undetectable samples, diluted samples and samples with results above the upper limit of detection for the assay. A few minor changes were suggested and Joan will finalize the document to include those minor changes. The table will then be given to FSTRF for implementation into the LDMS. 4) Memo: VQA Specimen Dilution Panel A final document was created for the specimen dilution plan that included comments from the last VQAAB conference call. The committee approved (8-0) the document (with a few minor revisions) and agreed that it should be sent to the laboratories for implementation. The plan would also be sent to the cross network Laboratory Focus Group (LFG) for informational purposes. The VQA will finalize and distribute the memo to the LFG and testing laboratories. The VQA will create a spreadsheet for use in capturing the data for the dilution validation as proposed in the memo. The VQA will create a report that will indicate how the data generated by the testing laboratory compared to the preliminary data generated by the VQA. The VQA will share the blinded reports with the VQAAB as they are created for testing laboratories. 5) Memo: Manual Abbott Validation A final document was created for the manual Abbott validation plan that included comments from the last VQAAB conference call. The committee approved (8-0) the document (with a minor revision) and agreed that it should be sent to the laboratories for their records/information. The plan would also be sent to the cross network Laboratory Focus Group (LFG) for informational purposes. The VQA will finalize and distribute the memo to the LFG and testing laboratories. 6) Proposed Changes for HIV RNA PT Scoring Carrie Wager presented the proposed changes for HIV RNA PT scoring during the transition from a citrated plasma matrix to a SERUM+EDT matrix. The current scoring criteria include accuracy, precision, sensitivity, specificity and sample validity. Of these 5 criteria, only sensitivity, accuracy and precision are most susceptible to being affected by a matrix change. The current procedures for scoring include combining four 5 member panels (or a combination of a 20-member prequalification panel plus five-member real-time testing panels) and assessing the five scoring criteria. This combining of data was necessary to permit between run precision analyses and to accumulate sufficient numbers of samples within the analysis to permit reasonable power for these analyses. With the phasing in of a new matrix, there is a change in the n of samples that can be combined for a given analysis which can affect the reliability of that analysis. Therefore, NERI put together a Created on 8/13/2010 1:07:00 PM 2 of 5
Created on 8/13/2010 1:07:00 PM 3 of 5 July VQAAB Minutes proposal that would address the issues that could affect VQA HIV RNA PT scoring. These changes are described below. Accuracy accuracy is assessed by comparing the average log10 recovery within a laboratory to the expected log10 recovery. The average laboratory log10 recovery is calculated using data generated by that testing laboratory over the last four rounds. The expected log10 recovery is estimated as the median log10 recovery across all laboratories in which the same kit was used on the panels in question. An algorithm is then applied that generates an accuracy statistic. This statistic is then given a score based on the scoring criteria. In the proposed modification, a weighted average of the differences between the average log10 recovery and the kit-matrix-specific medians will be employed. Simulator studies showed good consistency between these two methods of scoring suggesting the VQA will still be able to adequately assess accuracy as the new matrix is phased into the program. Precision the total assay standard deviation is calculated by combining the inter-assay and intraassay variation and calculating the standard deviation from that sum of variances. As the SER+EDT matrix is phased in there will be changes that occur that will adversely affect these precision analyses. In the first analysis, there will be data from 3 five member panels that were made in citrate and data from 1 five member panel that was made in SER+EDT. This reduces the n of samples available for inter-assay precision studies and there is no inter-assay variance for the SER+EDT data. For the second round of testing, there will be data from 2 five member panels created in citrate and 2 five member panels created in SER+EDT. For the third round of testing, there will be data for 3 five member panels created in SER+EDT and only 1 five-member panel created in citrate. By the fourth round of testing, the data for precision analyses will be a complete set and scoring could proceed normally for precision statistics. Simulations of precision analyses that separated the panels by matrix showed instability with respect to total assay standard deviations. The inter-assay variation component was noted as the main cause of this instability, most likely due to the small sample size in the analysis. Therefore, it was proposed that the VQA continue to monitor and score on intra-assay standard deviations, and to document inter-assay scoring for informational purposes only, until a total of 4 rounds of SER+EDT testing has been completed. Sensitivity sensitivity is assessed by determining the number of negative results obtained from replicates at the limit of detection for the assay in question. This limit of detection varies by kit and is defined as the nominal concentration at which 95% of the assay results are expected to be detectable. Preliminary studies performed by the VQA have shown that the current sensitivity challenge for the UltraSensitive Roche COBAS Amplicor Monitor test were not affected by the switch from citrate to SER+EDT. However, it s not clear how the matrix change may affect the results obtained on samples with a nominal value of 50cp/mL on the Abbott RealTime (AR) or Roche TaqMan (RT) assays. The VQA had started to do evaluations on the performance of this sample created in citrate, but there are insufficient data for samples with this nominal value in SER+EDT. The VQA proposes to continue to score sensitivity in kits that were currently being scored for sensitivity, but to hold off on sensitivity scoring for the new real-time PCR kits (AR and RT). These data will continue to be provided for informational purposes only but will not be included in the proficiency testing scoring. These proposed changes were discussed and approved (8-0) by the VQAAB members. The scoring changes will be implemented in the upcoming analysis of RNA023. The first five samples of that panel will be due 30 July. Scoring will revert back to the original scoring criteria after 4 rounds of
testing have been complete which will be the analysis that includes samples through RNA023xx.16-20A which are due 31 January 2011. 7) Review of New VQA Proficiency Panel Results. Suzanne presented the second report on the analysis on VQA HIV RNA proficiency testing data through samples 06-10A of RNA022. Four data sets from 4 laboratories were included in this analysis. Technical problems were identified in the one data set generated with the Standard Roche Monitor test. An elevated total assay standard deviation was found in this data set. The elevated precision statistic was caused by problems with intra-assay variation. A score of PC was recommended for this data set. No technical problems were noted in the one data set generated with the UltraSensitive Roche Monitor test. A score of C was recommended for this data set. Technical problems were identified in the one data set generated with the Roche TaqMan assay. An elevated total assay SD was found in one data set. The elevated precision statistic was caused by problems with inter-assay variation. A score of P was recommended for this data set. No technical problems were noted in the one data set generated with the BioMerieux EasyQ assay. A score of C was recommended for this data set. One laboratory received a score of NO DATA for both their Standard Roche Monitor assay and their Abbott Realtime data set. No data or request for an exemption were received by the extended due date for either data set. A score of ND is equivalent in point value to a score of P. These data were reviewed and approved (8-0) by the VQAAB committee members. 8) Change in Laboratory Status Report The change in status tracking sheet was circulated to the committee in a blinded manner for review. The scores in the tracking sheet reflect all the analyses done to date. For HIV RNA, there were 6 laboratories that would receive change in status letters as a result of this most recent analysis for RNA: 1 laboratory was REAPPROVED, 1 laboratory was NEWLY APPROVED, 3 laboratories were PROVISIONALLY APPROVED, and 1 laboratory was NOT APPROVED. An un-blinded tracking form will be created and sent to the cross-network leadership for review. Change in status letters be drafted by the VQA, reviewed by Bill Meyer and then sent to the laboratory PI, the affiliated network leadership and DAIDS. 7) Update on External 200cp/mL control The VQA has been trying to compile the data for the VQA 200cp/mL control, but the n of data is too small to do an analysis yet. The problem appears to be that many laboratories are not currently using the new real-time PCR platforms (AR and RT) for routine testing, which means they are only running the VQA control for proficiency testing, which occurs every two months. The VQA is going to work with FSTRF to collect all the extra control data that is available and will send the results of the analysis to this committee for review as soon as it is available. 9) New Items of Interest from the VQA -- The VQA had no new items of interest to discuss. Created on 8/13/2010 1:07:00 PM 4 of 5
10) New Items of Interest from the Committee Members Bill Meyer raised the issue of possibly reducing the number of HIV RNA PT challenges per year based on the observation that more laboratories seem to be doing better. He proposed reducing the number of challenges from six to four per year. The committee felt this would require more time for discussion so this will be added to the next month s agenda. However the VQA did want to suggest that any changes in frequency not change until after the new matrix has been fully phased in which would occur after January 2011. The next conference call is scheduled for Tuesday, 10 August 2010 at 1:00 PM Eastern. Created on 8/13/2010 1:07:00 PM 5 of 5