Rigorous Bench Studies to Determine Sensitivity & Precision of Athelas Strip & Device. At-Home Usability and Interpretability Studies for Consumers

Transcription

1 Rigorous Bench Studies to Determine Sensitivity & Precision of Athelas Strip & Device At-Home Usability and Interpretability Studies for Consumers

2 INTRO Current state of technology indicates that the gold standard mechanisms used by Labcorp, Quest Diagnostics, and Theranos produce results outside clinically defined ranges 8.3%, 7.5%, and 12.2% of the time for LabCorp, Quest, and Theranos, respectively. 1 These data points carry implications on precision and accuracy - namely, it is possible to get gold standard venous CBC results with +/ % differences between the same sample if run twice. As such, a determination was made during the Athelas Clinical Study that the existing gold standard in-lab venous precision was not the most stringent way to characterize accuracy. Additional bench studies were conducted with synthetically concentrated cellular samples of target tranches. Furthermore, consumer-grade usability studies were conducted to highlight the accuracy of the end-to-end device and strip usage in a raw, at-home setting. These studies focused on the accuracy and precision of the model when used by a completely untrained individual with no background. This study found that the Athelas system produced n=0 results outside of clinical range, maintaining reproducibility of results over 20 replicates, and on multiples devices. Further, when users (n=37) attempted to use the Athelas system with only the in-app manual, and paper manual present, all users were able to produce a result that was clinically significant, benchmarked with the Labcorp results generated by a clinician indicating that the Athelas device is understandable and easy to use by an untrained individual. BACKGROUND The purpose of these bench studies is twofold: to exhaustively determine the sensitivity of the Athelas device in various cohorts, and to determine the usability and interpretability of the device when tested in home with only the in-app instructions and the paper manual. Often in diagnostics, accuracies drop when going from clinical studies and in-lab data to the real world with real users. To assess and characterize the performance of the Athelas platform in this end-user setting, Athelas organized a trial focused on patients seeing the device for the first time. 1 J Clin Invest. 2016;126(5): doi: /jci

3 METHODS Bench studies were conducted to test the precision of the test strips, inter device precision, and sensitivity. Additional at home studies were also conducted across a range of demographic groups to characterize the usability of the device and strips and the interpretability of the results. The clinical gold standards2 were in many ways deemed unfit to fully characterize and define the accuracy of the Athelas system due to their own inherent variation. As such the Athelas study made use of synthetic PBMCs3 with tranched concentrations to synthetically create the test samples. Table 1. Collected by B. Kidd in Evaluation of direct to consumer low volume lab tests in healthy adults. J Clin Invest. 2016;126(5): doi: /jci86318 J Clin Invest. 2016;126(5): doi: /jci Kirchner, H., Ch Kleinicke, and W. Digel. "A whole-blood technique for testing production of human interferon by leukocytes." Journal of immunological methods 48.2 (1982):

4 Bench Study - Sensitivity, Precision, Aging The objective of this study was to establish the measurement accuracy stability of the Athelas system as samples aged. A parent vial of 10^8 white blood cells was diluted to create solutions of seven different concentrations (2k, 4k, 5.5k, 9k, 11k, 20k WBC/uL). 4 ul of solution 1 was loaded into 20 test strips. After 5 minutes, each strip was inserted into the machine and a result was obtained and recorded. The same strips were inserted into a second machine and another set of results were obtained and recorded. The procedure was repeated for each solution. At Home Testing - Usability & Interpretability The objective of the study was to demonstrate the accuracy and interpretability of the Athelas System when used by patients. The patients were given the Athelas app (Figure 1) and an instruction manual. Figure 1. Athelas in app instructional gif The patients were asked to perform their own tests from start to completion with no assistance from the Athelas team. The patient's WBC count was compared using the Athelas device versus the results generated by the LabCorp system (Beckman Coulter UniCel DxH 600/800 COulter Cellular Analysis System). Additionally, the patient was asked to report how easy it was to understand the Athelas instructions (Table 2) and the final results (Figure 2). 4

5 A. Did the user understand where the strips were located? B. Did the user understand how to fill the strip? C. Did the user fill the test strip properly? D. Were they able to place it in the correct slot in the device? E. Are they able to perform a successful test? F. If no, was it clear to the user that the test was unsuccessful? G. If no to 1, was it clear that the user should perform another test after their unsuccessful test? H. Did the user perform another test after their unsuccessful test? I. How many times did the user perform a test to ensure a proper reading? Indicate agreement on a 1-5 scale (1=strongly disagree; 5=strongly agree). 1. The instructions were easy to follow. 2. It was easy to apply the sample correctly. 3. It was easy to see and understand the test results (e.g., appearance of the line, change of color). 4. The instructions clearly explain what to do if a test result does not appear or is invalid. 5. I needed help from someone the first time I ran the test. 6. If I did not perform the test properly, I was sure that I had to repeat it. 7. It was clear that this test did not provide a diagnosis. 8. If I performed the test correctly, I knew that I had done it correctly. 1. How difficult is it to read the Labcorp results 1. How difficult is it to read the Athelas results What was the Labcorp Count? What was the Athelas Count? Table 2. Athelas questionnaire to determine patient usability and interpretability. The study was conducted at 275 Moffett Blvd. Mountain View, CA and LabCorp (LabCorp San Jose, 266 N Jackson Ave Ste 4 San Jose, CA 95116). The study plan and proposal was approved by the institutional review board (IRB) at Salus IRB. All participants signed an informed consent form that detailed the objective of the study, the sponsor of the study, the procedures of the study, the right to opt-out, the anonymization of results, and how their anonymized results may be used. 5

6 RESULTS Results from Bench Studies The method used for the bench studies supports that the Athelas system generates accurate results throughout the indicated range (2k-20K WBC/uL) and in the vicinity of cut-off thresholds (Figure 3) with minimal confusion in the border ranges, particularly in the ranges between 4K WBC/uL and 5.5K WBC/uL. This confusion can most likely be attributed to natural (<10%) variation in WBC concentrations. More rigorous testing precision will be done to validate this result. Athelas WBC Consistent Over 20 Trials Across 2K-22K WBC/uL Figure 3. Athelas Bench Study Data conducted at Stanford University, collected on April 3, The methods used has been demonstrated to be linear between 2k/uL-20k/uL within no different in 2k/uL and within no difference in 20k/uL. Setup, takedown, and testing at different times does not affect the results generated by the system. The results also indicate that testing the strip on different devices yields a correlation coefficient > r^2=.8 (Figure 4). 6

7 Inter Device Precision Over 20 Trials Across 2K-22K WBC/uL Figure 4. Athelas Bench Study Data conducted at Stanford University, collected on April 3, Results from At-Home Testing Of the 37 users, aged (51% female, 49% male), three male users (age=17, 69, 71) had to redo the test as they received the N/A screen and 34 users were able to generate their results on their first try. Of the three users who were not able to perform the test on the first attempt, all of them indicated they had done the test incorrectly. Upon second attempt, the three users were able to generate their results. The results generated by the users had an r^2 = with the results obtained by Labcorp for their corresponding sample (Figure 5). 7

8 Labcorp WBC Count vs. User Generated Athelas WBC Count Figure 5. Results of Athelas At-Home Testing conducted by users. In regards to interpretability of results, all 37 individuals in the study were able to correctly identify the different components of their Labcorp blood report accurately upon first attempt, and all 37 individuals in the study were able to correctly identify the different components of their Athelas report on the first attempt. Based on the answers of the users, they believed the interpretability of a Labcorp result to be (4.4)/5 and the interpretability of the Athelas test to be 5/5 with 5 being the easiest to interpret (Table 2). 8

9 Athelas Usability Questionnaire: Answers by Untrained Users 9

10 Athelas Usability Questionnaire: Answers by Untrained Users Table 2. Athelas answers to usability questionnaire. Completed by users 10

11 Conclusion Quality measures of the gold standard mechanism done by Mt. Sinai indicate that the current gold standard technology used by Labcorp generates results for a host of tests, including a WBC count, out of clinical range (0-3.4K WBC/uL, 3.4K-10.8K WBC/uL, >10.8K WBC/uL) in 8.3% of cases. 4 The Athelas system produced n=0 results outside of clinical range, while also maintaining reproducibility of results on 20 replicates of strips, and reproducibility of results on multiple devices. The Athelas system also proves to have developed a substantially equivalent platform to understand erroneous test procedure, and interpretable results as compared to the gold standard. When users attempt the Athelas system with only the in-app, and paper instruction manual, all 37 randomly selected users (age 10-73) were able to generate clinically significant results, benchmarked with the Labcorp results generated by a clinician. 4 J Clin Invest. 2016;126(5): doi: /jci

12 References B. A. Kidd, G. Hoffman, N. Zimmerman, L. Li, J. W. Morgan, P. K. Glowe, G. J. Botwin, S. Parekh, N. Babic, M. W. Doust, G. B. Stock, E. E. Schadt, J. T. Dudley, Evaluation of direct-to-consumer low-volume lab tests in healthy adults. J. Clin. Invest. 126, (2016). Kirchner, H., Ch Kleinicke, and W. Digel. "A whole-blood technique for testing production of human interferon by leukocytes." Journal of immunological methods 48.2 (1982):