Early-Stage Non Small Cell Lung Cancer: Quantitative Imaging Characteristics of 18 F Fluorodeoxyglucose PET/ CT Allow Prediction of Distant Metastasis

Transcription

1 Herbert Y. Kressel, MD Hi. This is Herb Kressel and welcome to the October Radiology podcast. This month we have three discussions for you. First, I ll be speaking with doctors Ruijiang Li and Daniel Rubin from Stanford who wrote a very provocative piece on Early-Stage Non- Small Cell Lung Cancer: Quantitative Imaging Characteristics of FDG PET/CT Predict Distant Metastasis. Lots of interest these days on biomarkers that can be used for prognosis and I think our readers will find the discussion quite stimulating. Next, my colleague Dave Kallmes, Deputy Editor for neuroradiology will be speaking with Dr. Eyal Lotan who with his colleagues at the Tel-Hashomer Hospital in Israel reported on the diagnostic value of apparent diffusion coefficient for the accurate assessment and differentiation of postoperative intracranial abscesses. This is an unusual application of the fusion imaging and we think you ll find this discussion of interest. And then finally, I ll be speaking with Dr. Matthew McInnes of the University of Ottawa who has an interest in meta-analyses and with his colleagues there published a paper in our journal, Diagnostic Accuracy Meta-Analysis and Imaging Journals: An Analysis of Pooling Techniques and their Impact on Summary Estimates of Diagnostic Accuracy. The short form is that not all meta-analyses of diagnostic accuracy are the same, and depending on how you analyze them the same papers can have surprisingly different results. So we hope you enjoy this month s podcast and we look forward to any feedback that you have. Early-Stage Non Small Cell Lung Cancer: Quantitative Imaging Characteristics of 18 F Fluorodeoxyglucose PET/ CT Allow Prediction of Distant Metastasis Radiology 2016; 281: Jia Wu, PhD Todd Aguilera, MD, PhD David Shultz, MD, PhD Madhu Gudur, PhD Daniel L. Rubin, MD, MS Billy W. Loo, Jr, MD, PhD Maximilian Diehn, MD, PhD Ruijiang Li, PhD Herbert Y. Kressel, MD Hi. This Herb Kressel editor of Radiology and welcome to this month s podcast. Today I m joined by Dr. Ruijiang Li, Assistant Professor of Radiation Oncology at Stanford University and Dr. Daniel Rubin, Associate Professor of Radiology at Stanford. These two are among the authors in a very provocative study entitled Early-State Non-Small Lung Cancer: Quantitative Imaging Characteristics of FDG PET/CT to Predict Distant Metastasis. Welcome Dr. Li. Ruijiang Li, PhD Hi. HYK Hi. Welcome Dr. Rubin. Daniel L. Rubin, MD, MS Hello, thank you for having us. HYK Dr. Li let s begin with you. What was the rationale for this study? Why did you and your colleagues undertake it? RL Well I think the clinical motivation is that over the coming years there will be increased prevalence of early-stage lung cancer, the success of screening programs. A key issue in managing patients is that while surgery or radiation therapy especially with sterotactic therapy, we have external local control rates. But some fundamental issues still that significant (inaudible) who have distant metastasis and that s the predominant cause of death in these patients. Our goal is to identify which of these patients will have high risk or distant metastasis so that we can potentially target them with adujuvant therapy which might increase their survival. HYK I see. So that would effectively serve to make screening efforts of greater value by also treating those most likely to have distant metastasis. Can you tell us what you did in this study Dr. Li? RL Sure. We looked at the FDG PET scans for these patients. For patients getting radiotherapy we routinely get FDG PET and CT for treatment planning purposes. We retrospectively looked at about 100 patients treated at our clinic and these patients get they all got uniform stereotactic radiotherapy treatment. And then we used a quantitative radiomic approach to extract a set of quantitative image features from FDG PET scans and then we used statistic approaches to develop an imaging signature which will hopefully predict distant metastasis. HYK Thank you. As I read this study one question I had was actually was PET CT that was performed on all patients, so why not analyze the CT image features as well as the PET features and moreover, motion is more of an issue in terms of the PET data collection since it s sort of longer acquisitions so this would at first blush appears to be sort of less than optimal approach to the analysis. Why didn t you look at the CT? Dr. Rubin you want to start? DLR: Sure. I ll start and Dr. Li can talk about the PET motion. That s an excellent question. Certainly the PET

2 data comes with the CT. One thing you don t generally have on board with the CT on a PET CT is injection of intravenous contrast. At our institution a chest CT is done at much thinner slice thickness than the PET CT. Certainly you can see CT characterization of lung nodules with quantitative methods, it s desirable to have very, very thin slices for optimum spatial resolution. However, it is certainly possible and valid question asked well we could have certainly have tried to pull out quantitative features from the CT and that would certainly be worthwhile looking at in future work because operationally if there s value in it it would be nice if you just did it as part of one test as opposed to two separate tests. Now one of the reasons why there was focus on the PET as opposed to CT as the first study to go for, is the PET of course being based on a physiological radiopharmaceutical offers the promise of quantitative physiological characterization which is where we believe a lot of the signal in characterizing lung cancer lies. Also, there have been a number of papers already looking at quantitative features in CT, in the Radiology journal actually. One paper from our group and others as well that have looked at that so we wanted to focus for novelty on the PET, but to your point I think it would definitely be worth looking at the CT in the future. HYK Dr. Li any more thoughts on sort of the issue of the motion effects on PET and they ve affected the data? RL Sure. Actually if you look at population level there s studies that investigate the degree of lung tumor motion, so on average to get about 5 mm motion for average patient and one key difference from diagnostic scans in these patients is that because they re getting radiotherapy, motion is a really important factor. Something I would like to link, so in all patients getting (inaudible) simulation scans including these PET scans, we have routine measures to limit the amount of motion such as shallow breathing or abdominal compression, all the things that we do to basically to limit the amount of motion here in scans. Ideally we have the 40 PET scans where the motion basically is motion free scan, but the issue with that is the PET scans has a reduced Cox if you ve further been different data from different breathing phases. There s an increased signal-to-noise ratio (inaudible). On the other hand there s reduced signal on the motion free scans. HYK Got it. So can you highlight your key findings because you developed a quantitative metric, a model that might be useful in predicting those patients who will develop distant metastasis. Can you tell us the key findings of your study? RL Yes, so we extract about 70 features then we basically applied two rules to further (inaudible) these features. One is that this feature, we had to be not redundant because usually as you re looking at a variety of features, there can be correlation among them. So we first require them to be not redundant. The second because there s an uncertainty, intrinsic uncertainty regarding to the delineation of tumor, so the second (inaudible) implies that this feature has to be robust against variation in different simulations. So with that we have, left us with about half or 35 features that we used to develop a statistical model. Still that s 35 a relatively large number compared to the number of patients we have. We now used a lasso technique which can help us to perform feature selection and at a same time to optimize the model. In the end we end up with two image features. One of them is peak SUV which is well characterized in many studies. It s similar to SUVmax but it s been shown to be more robust. And the other one is what s called texture features that describe the intratumeral heterogeneity of metabolic activity. Two features contribute together to determine metastasis risk. HYK Could you just briefly tell us what the lasso technique is? I read it in the paper. I m not familiar with it. RL So lasso basically is what statistically speaking is regularization technique. Because you have a large number of features, lasso technique will require you use a very small number of features to free the data. Without lasso you have all the (inaudible) but in lasso you end up with only a few. HYK It helps you narrow down. Dr. Rubin our statisticians very commonly as a rule of thumb suggest ten subjects per variable or features that are analyzed in this type of study. In this one you did another technique one of these leave one our analysis, again I think is aimed at dealing with the issue of avoiding the problems of multiple comparisons. Can you tell us a little bit about that approach and how does it relate to the lasso business? DLR So the lasso part of the technique is the statistical model for making predictions given the input covariates. The cross validation technique is what do you do to train the model and test the model once you ve established the model. So that s a model that s established as Dr. Li explained. Now what we need to do is there are three parameters that need to be fit. So to fit that you re going to need to take a set of training cases and compare, you ve run them through the model and compare the model output with the ground truth that was known from those training case and adjust the weights intervally until you get a good fit. Of course you don t want to test the model you ve just fit on exactly the same data that you trained with because you ll be subject to a phenomenon called overfitting and the result won t be reliable for testing on unseen data in the future. Generally one does a procedure called cross validation. We use a variant called leave one out where you hold out, in this case leave one out, one case and you build a model on the remaining cases

3 and then once you fit the model you test it on the held out case, so the held out case didn t participate in training and therefore is valid for testing. And then you repeat that multiple times for each of the other cases. You generally pick a leave one out over holding out some large number when the number of cases you have isn t substantial. Ideally, we would have, your statistician s point is well taken and ideally you d like to have as much data as possible. We re limited by the data available and in the future would like ourselves to either require more cases, or other investigators preferably repeat our study and find out that they come up with similar results to make this most reliable. But at least for the data we tested with, leave one out cross-validation is a valid procedure for evaluation. HYK Great. Now this is I thought really exciting. It s sort of the direction that we really all want and need to move to get better quantitative parameters to help mitigate undesired outcomes. What are actually the next steps in the research you two are undertaking? Dr. Li what is next steps in this line of research? RL I think there are a few directions going forward. One of them is that we d like to explore the underlying mechanism of these imaging features. Radiomic approach is really a high throughput diagnostic approach without knowing what really what it means. So I think next we can potentially integrate imaging with genomic data to see what molecular features correlate with the imaging findings from the study. Of course the other step is that really we need to validate this imaging marker in large cohort, but preferably in multi-center perspective studies. HYK Got it. RL The other one as Dr. Rubin mentioned looking at CT features and see where that complements what we find is another direction. HYK Dr. Rubin any other thoughts on next steps in this line of research? DLR Yes I would say one of the issues of quantitative imaging in terms of clinical acceptance to radiologists is the black box problem of there s this model, I don t completely understand it, there are these features that go and they came from an image that couldn t see the features that the model saw because they came from a computer. Can I believe it? It was done on one data set. At a minimum we would like to have this model applied on a second independent data set with cross validation, that was still one data set. That data set is necessarily representative of the entire population of patients that exist and certainly regionally, but nationally and internationally. For there to be believability on our part and others in terms of this model, we d like to at least have someone else whose got retrospective data try it out and see if they get comparable results. HYK Right. Now we haven t talked about this before, but one of the things that comes up when we see these studies is the issue of integrating clinical parameters into these imaging based models. Now I m sort of at the limits of my understanding, but I have been led to believe that including biological variables adds another level of complexity to the analysis and is not quite as straightforward as it might appear. Any thoughts about that Dr. Rubin? DLR Yes. The lasso model is a multi-variant model and as you point out, there are different ways of approaching integration of other kinds of data. The simplest approach would be to simply expand the input covariates to the lasso model. That will work pretty well with most clinical variables because those are discrete, small number of additional variables. Molecular data is a little more challenging. One approach could be we throw in all the gene expression but there s tens of thousands of genes and there s correlations between them. Now the lasso as part of a shrinkage operation throws out correlated variables, but in terms of the input signal, the signal from biological data may be the higher level than individual genes, maybe to develop pathways or pathways working together and so some kind of preprocessing or extraction is generally needed to integrate molecular data in with the clinical and imaging data. What many people do is they will do a technique called meta gene generation or will do gene set enrichment analysis, these kinds of things. You get abstractions from that very granular data up. One can simply add that into a lasso kind of model, but people can look at other models other than lasso, such as maximum entropy or is Bayesian-based models. By and large the models don t vary that much but until we have multiple people comparing these different things, their variables affect which we don t completely understand yet, and I m hopeful that other people will be studying these problems with those kinds of methods. HYK Right. Any final thoughts on this line of research Dr. Li? RL Well I think yeah the real motivation with these studies that we really want to have optimal way to manage these group of patients in early state lung cancer. We know that we can get very high local control rates with sterotactic therapy. Still there s a small portion who have distant metastasis. I think the optimal way is to combine all (inaudible) model because all these things influence these outcomes, clinical, (inaudible) or genomic level I think really move forward we need to discover new biomarkers and validate them in a large (inaudible) cohorts. HYK Very good. Well I want to thank you both for a stimulating discussion on a pretty complex topic. It s been a pleasure chatting with you. Thank you for participating. Bye-bye.

4 Postoperative versus Spontaneous Intracranial Abscess: Diagnostic Value of the Apparent Diffusion Coefficient for Accurate Assessment Radiology 2016; 281: Eyal Lotan, MD, MSc Chen Hoffmann, MD Alexander Fardman, MD Tomer Ziv-Baran, PhD Orna Komisar, MD Sagi Harnof, MD David F. Kallmes Hello. My name is David Kallmes. I m the Deputy Editor for neuroradiology. Today I am joined on this podcast by Eyal Lotan who is currently a fellow in neuroradiology at New York University and recently finished his residency training at Sheba Medical Center, Ramat-Gam, Israel. We are here today to discuss his paper entitled Diagnostic Value of Apparent Diffusion Coefficient for the Accurate Assessment in Differentiation of Postoperative Intracranial Abscess. Dr. Lotan, welcome to the Radiology podcast. Eyal Lotan, MD MS Thank you very much. Thank you very much for inviting me to talk about the study. DFK Our listeners may have read your study but just even so could you briefly tell us what you did in the study and what your major findings were. EL Sure. In the study we tried to assess the value of apparent diffusion coefficient, or ADC, for the diagnosis of postoperative intracranial abscess compared with spontaneous intracranial abscess. We defined an abscess as postoperative or spontaneous depending on whether or not a primary nonsurgical procedure was performed. This was a retrospective study and we included all the patients that had clinical and imaging data over a ten year period from 2005 to For each patient we used a dedicated workstation and we marked three region of interest on the ADC map including all the visible area of the abscess. On the upper and middle and lower planes and that s why we calculate the ADC value and we also evaluate the abscess qualitatively so with restricted diffusion and for the homogeneity of the diffusion pattern. So we had 73 patients, 70 patients with spontaneous abscess and 43 patients with postoperative abscess and we found that on DWI all the spontaneous abscess had a restricted diffusion and almost all like 80% of them had homogeneous diffusion pattern. On the other hand, postoperative abscess only 60% of them had restricted diffusion and only 30% had a homogeneous diffusion pattern. When I gathered the ADC value, the median ADC values of postoperative abscess was It s almost double the median ADC value of spontaneous abscess. DFK Did these findings, the fact that postoperative abscesses are not typical compared to the usual restricted diffusion with abscesses, did that finding surprise you or were you expecting it? EL Well yes and no. We all know that DWI is a very good method for the diagnosis of spontaneous abscess and we all use it all the time. But our clinical experience suggested that using DWI in the postoperative setting may be misleading and maybe this is a good place to give credit to my mentor and co-author Dr. Hoffman. We were surprised by the significant difference between the values. DFK Now can you give any pathophysiological mechanism that might underlie the differences in how the postoperative versus spontaneous abscesses are behaving? EL Okay a possible explanation for this difference may be depressed cellular immune responses for a patient with postoperative abscesses with lower inflammatory viable cell concentration maybe because of their underlying disease process or therapy. More explanation can be focal hemorrhage or postoperative fluid collection at the site of the infection or the use of intraoperative exogenous materials. Another explanation can be that the surgical intervention and also the underlying pathologic process can disrupt the normal anatomy and can lead to dead space. We ve compromised vascular supply so the resulting lower cellular and protein concentration at the site of the infection. DFK Those are all interesting postulates. Do you have any empiric data from drainages of the abscesses that might point you one way or the other in terms of cellularity, protein consent, concentration hemorrhage, etc. EL So we still don t have it because it was a retrospective study. We don t have it unfortunately. We know that the relationship between histology and restricted diffusion in brain abscess is complex and unclear, it s unknown. It is assumed to be due to combination of inflammatory viable cells and (inaudible) debris, viscosity, and macromolecular, concentration of macromoleculars (inaudible) in the past. But we hope that better understanding of this relationship will have it very soon. DFK This is a very interesting study and it certainly is provocative and I think it gets our attention that we shouldn t assume that any collection that s infected within the cranium should have restricted diffusion. But you didn t look at a group of patients who were postoperative that had sterile collections and so how can you apply these data in patients who have had a cranial procedure and present with suspected infection in the setting of a fluid collection? Can you extend your data to give the differential diagnosis there or is that the domain of a future study? To clarify, you ve shown nicely that it s not at all unusual for postoperative infections to have a restriction of diffusion that s unlike spontaneous abscesses. EL Yeah.

5 DFK However, the clinical question is going to be a patient comes in having had surgery with a collection. You want to know not whether it s spontaneous or postoperative because you know it s postoperative, you want to know is that postoperative collection infected or not. Does your study help or are future studies required to answer that question. EL Yes. Thank you. Our study didn t study this question because what we want to say that what they already say that also DWI is an excellent tool for abscess without surgery when we know that there was surgery DWI is not sufficient and shouldn t be used as the key modality in this situation and clinical judgment and clinical suspicion it needs to remain very important. DFK Right, I agree. I was asking a leading question. But it seems to me if a patient comes with a postoperative collection that doesn t have restricted diffusion, you should not exclude the diagnosis of infection, you should pursue it. So I think that is an important teaching point. We don t know the denominator in how many of those patients who are postoperative with an unrestricted diffusion collection are infected versus not, but we know that at least some of them are. EL Yes. As I said almost 40% of the patients without restricted diffusion in our study actually had proven diagnosis of abscess. DFK Yes. I think it s a very important question and it s a very clinically relevant answer you gave us. We think it will be immediately applicable in the clinic. I thank you for joining us. Before we finish, is there anything that we didn t cover that you want the listeners to hear about? EL I think we talked about the main issues of the study. DFK Thank you for your support of Radiology and we look forward to future excellent studies from your group. EL Thank you very much Dave, thank you very much. Meta-Analyses of Diagnostic Accuracy in Imaging Journals: Analysis of Pooling Techniques and Their Effect on Summary Estimates of Diagnostic Accuracy Radiology 2016; 281:78 85 Trevor A. McGrath, BSc Matthew D. F. McInnes, MD, FRCPC Daniël A. Korevaar, MD Patrick M. M. Bossuyt, PhD Herbert Y. Kressel, MD Hi. This is Herb Kressel editor of Radiology and welcome to this month s podcast. Today I m joined by Dr. Matt McInnes who is Associate Professor of Radiology at the University of Ottawa and who with his co-authors has authored a very stimulating study looking at the diagnostic accuracy meta-analyses that are published in imaging journals and analyzing the pooling techniques and their impact on summary estimates of diagnostic accuracy. We re seeing more and more studies of this sort and the work of Dr. McInnes and colleagues I think is very helpful in sort of peeling back the onion a bit to better understand the drivers of the estimates of diagnostic accuracy in these types of studies. Welcome Dr. McInnes. Matthew E. F. McInnes, MD, FRCPC Thank you Dr. Kressel. HYK So let s get started. What s the rationale for studying meta-analyses? Aren t these at the top of the evidence based medicine pyramid? They re already at the top of the food chain. What do we need to study these for? MEFM Well I think meta-analyses like any other research can be done well or they can be done not so well. And certainly when a meta-analysis is done well and it s based on high quality evidence it can be at the top of the research food chain pyramid or certainly near it. But when it s done is ways that are sub-optimal it might not be reflective of true results and may be more misleading than anything because people may put a false sense of authority or put it up at the top of the hierarchy when it s not necessarily appropriate. HYK I see. Now in your study your focus was on imaging journals. Are these different than imaging studies rather, are these types of studies different in any way? Do they present some particular challenges in meta-analysis? Why did you choose to study these meta-analyses? MDFM Well the short answer is I studied imaging journals because I m a radiologist so it s of interest to me, but the particular challenge of imaging is that we re often interested in test accuracy so measures of sensitivity and specificity. Now when you do pooling in meta-analyses or you combine the results from multiple studies, typically you use what s called a univariate random effects model which will just pool the results on their own not considering anything else. If you do this with sensitivity and specificity it ignores the correlation between those two factors and it can potentially lead to misleading results so we need special tests when we re dealing with diagnostic accuracy measures called bivariate models or hierarchical models and if you don t use those it can potentially lead to misleading results. So imaging in and of itself is of interest to me. There are some unique features that we have to consider in imaging studies. HYK So my understanding is that the recommendation to use these bivariate or hierarchical types of analysis are

6 relatively recent. When were these introduced and was this specific for imaging studies or for a range of studies? MDFM These were not specific for imaging studies. It s for any diagnostic test accuracy study whether it s a lab test, physical examination, imaging test there s a whole bunch of diagnostic test accuracy tests that you can do, but imaging is certainly a large component of those the hierarchical co-model by Rutter and Gastonis was introduced in 2001 and the bivariate model by Reitsma and his group was introduced in Like with any research, that can often take a long time to disseminate and certainly the number of meta-analyses being done in imaging has really been exploding of late, but the knowledge on how to do them has taken a while to disseminate. The Cochran diagnostic test accuracy methods group recommends that we use those models. That recommendation started about 2009, but it s taken a while to get out there because as imaging researchers we may not be aware of that recommendation. So part of this study is to raise awareness of proper methods for pooling. HYK I see. What did you do exactly in your study? Can you tell us about your study design and methods? MDFM Sure. So we wanted to evaluate whether meta-analyses done in imaging journals were using appropriate hierarchical pooling techniques when they re looking at diagnostic accuracy measures. So we looked at about 300 meta-analyses published over a ten year period from 2005 to 2015 in imaging journals and that s loosely categorized and defined by Thompson and Reuters so we just used that group of journals. So of these 300 we looked to see whether the authors used bivariate or hierarchical methods or they used traditional univariate methods which wouldn t generally be recommended for diagnostic test accuracy meta-analyses. What we found was that a minority of journals were using the appropriate bivariate or hierarchical methods so less than 40% were using them and then what we did is we took the ones that the studies that used a univariate method or the method that s not recommended and we reanalyzed their results using the appropriate method using a program called R. When we reanalyzed the results we looked to see whether there were differences in the sensitivity, specificity or the confidence intervals derived from one method versus the other and we did find that there were differences. So the differences we found were that the traditional or the univariate models that are not recommended tend to overestimate diagnostic accuracy with more narrow confidence intervals. The overestimation is not huge. It s about 2% for sensitivity and specificity and the confidence intervals are about 7% more narrow than the proper bivariate model were derived. Now some people might say 2% isn t a lot, but a proportion of the studies had large deviations. So deviations of 5 to 10 percent in the summary estimates were seen in 7 of the 50 studies we looked at for sensitivity and 11 of the 50 studies for specificity. So these are some pretty huge deviations we re seeing in about 20% of the studies. So there s potential for a lot of misleading data that we put out there. HYK What about the mean 7.7% change in the width of the confidence interval? Do you think that s kind of clinically significant in general? MDFM I think so. I mean it has to do with how certain a radiologist would be in the range of the results when applying a test and it might have importance in let s say guidelines, organizations, or governmental organizations that might recommend a test on a population basis and they might use the confidence intervals to estimate in models on how the test might perform. And if those confidence intervals are too narrow, they might come up with some misleading data and the results might not be reflective of truth. So I think that broader scale when meta-analyses are often used to pump data into cost effectiveness models at the population level, those more narrow confidence intervals might have a huge impact. HYK I see. You looked at number of different imaging journals were there any important differences in the views to the hierarchical approach in the different imaging journals? MDFM There were. So we looked at journals that had published more than ten meta-analyses within our sample and there s quite a bit of variability between the journals. Journals like Radiology and European Radiology that are sort of more aware in promoting these models had good but not great uptake with about 50 to 75 percent of their studies using the bivariate models. Some other journals where the awareness isn t as high had lower estimates. So quite a lot of journal level variability which certainly speaks to the fact that the awareness of these proper methods is not disseminated uniformly and authors, reviewers, editors, really need to be aware of these techniques, but the optimal techniques are in order to ensure that we re publishing good data. And mea culpa the first meta-analysis I published in Radiology actually didn t use a bivariate model and this was around the time where Cochran was figuring out what was the optimal model and I discussed it with the statistician and he said oh no we don t need a bivariate model so I m one of the offending authors. Part of this study was derived from my own experience. HYK In all fairness, the recommendations were later than the start date of your study accrual. MDFM Correct, correct. HYK 2009 but you were accruing from What about meta-analyses in different areas of imaging. Were there important differences in different in sub-specialties? MDFM There were. Again there was quite a lot of variability between these sub-specialties. It s hard to know

7 what these differences mean, but gastrointestinal and obstetrical sub-specialties had better use or higher levels of use. That may be driven by a few centers and a few authors in areas like the Netherlands where these techniques were developed and disseminated. So maybe more of just a geographic or institutional effect where a few researchers are doing a lot of work in these areas and therefore are representing a lot of work in those sub-specialties. But again I think the variability speaks for opportunities for more wide, broad dissemination for what the appropriate methods are. HYK So for many of our viewers and listeners, this talk may seem somewhat esoteric we re in the weeds of this stuff, but what do you think are the important take home message of your study for readers of these types of papers? MDFM I think readers need to be aware that there are specific techniques needed for pooling of data and diagnostic test accuracy studies. If they don t see the words bivariate or hierarchical models in the description of the results, they should be circumspect about whether appropriate techniques have been used. We hope that with this study and others the awareness is raised and that we ll see less and less of the univariate studies being used. However there s a free-ware program out there called Meta-DiSc which advertises itself as appropriate for DTA meta-analysis and unfortunately it doesn t use the appropriate techniques. So this is one of the it s actually a very nice program, it s well written and it works really well, and until recently authors hadn t had an easy program to use other than that. So fortunately R now has a software package you can use and it s freely available, but we re swimming against the tide a bit because a lot of authors just Google DT meta-analysis, find this program, and it s there. So in short I think if authors are aware we need bivariate methods and they see that, they can be assured that at least the pooling has been done in a manner that s appropriate. HYK Very good. Dr. McInnes I want to thank you very much for this paper and for bringing this important issue to the floor and it s been a pleasure talking with you. MDFM You re welcome. Have a great day Dr. Kressel. HYK You too. Take care. Bye-bye.