Regression Benchmarking with Simple Middleware Benchmarks
|
|
- Rudolf Anderson
- 6 years ago
- Views:
Transcription
1 Regression Benchmarking with Simple Middleware Benchmarks Lubomír Bulej 1,2, Tomáš Kalibera 1, Petr Tůma 1 1 Distributed Systems Research Group, Department of Software Engineering Faculty of Mathematics and Physics, Charles University Malostranské nám. 25, Prague, Czech Republic phone , fax Institute of Computer Science, Czech Academy of Sciences Pod Vodárenskou věží 2, Prague, Czech Republic phone {lubomir.bulej, tomas.kalibera, petr.tuma}@mff.cuni.cz Abstract The paper introduces the concept of regression benchmarking as a variant of regression testing focused at detecting performance regressions. Applying the regression benchmarking in the area of middleware development, the paper explains how regression benchmarking differs from middleware benchmarking in general. On a real-world example of TAO, the paper shows why the existing benchmarks do not give results sufficient for regression benchmarking, and proposes techniques for detecting performance regressions using simple benchmarks. 1. Introduction The development process of a software system is typically subject to a demand for certain level of quality assurance. One of the approaches to meet this demand is regression testing, where a suite of tests is built into the software system so that it can be regularly tested and potential regressions in its functionality detected and fixed. The complexity of the middleware has led many middleware projects to adopt some form of regression testing, as evidenced by open source middleware projects such as CAROL [1], OpenORB [3], or TAO [7] with its distributed scoreboard [6]. Focusing on functionality, however, the regression testing of middleware tends to neglect the performance aspect of quality assurance. With the notable exception of middleware projects that provide real-time or similar quality of service guarantees, performance is typically orthogonal to correct functionality and thus seen as a minor factor in quality assurance. This contrasts with the otherwise common use of middleware benchmarking to satisfy the obvious need to evaluate and compare performance of numerous implementations of middleware. To remedy the existing neglect of the performance aspect in regression testing, we focus on incorporating middleware benchmarking into regression testing. Our experience from a series of middleware benchmarking projects [4][5][11][13] shows that systematic benchmarking of middleware can reveal performance bottlenecks and design problems as well as implementation errors. This leads us to believe that detailed, extensive and repetitive benchmarking can be used for finding performance regressions in middleware, thus improving the overall process of quality assurance. For obvious reasons, we refer to such middleware performance evaluation as regression benchmarking. In section 2 of the paper, we investigate the concept of regression benchmarking, explaining why and how it differs from benchmarking in general. Section 3 illustrates why the existing benchmarks do not give results sufficient for regression benchmarking, and proposes guidelines and techniques for detecting performance regressions using simple benchmarks. Section 4 outlines the future work and concludes the paper. To illustrate the individual points and proposed techniques, we use TAO [7] as a real-world example of a complex and mature middleware. Two benchmarks are used throughout the paper. Denoted as benchmark
2 A is a benchmark that measures the duration of a remote method invocation with an input array of 1024 unsigned long values, denoted as benchmark B is a benchmark that measures the duration of marshaling an input array of 1024 unsigned long values. All results were collected on Dell Precision 340 Workstation with Pentium 4 2.2GHz and 512MB RAM running Linux with GCC Regression Benchmarking Regression benchmarking is a specialized application of benchmarking as a method of performance evaluation that is tightly integrated with the development process and fully automated. By integrating the regression benchmarking framework with the middleware development framework, new regression benchmarks can be added alongside new middleware features. The integration minimizes the cost of creating and maintaining regression benchmarks and has the added benefit of the benchmarks supporting the same platforms as the middleware. The regression benchmarks must be fully automated so that they can run unattended. The requirement of automation concerns not only the execution of the benchmarks, but also the data acquisition and the results analysis. The automated execution appears to be a simple task, with the existing remote access and scripting mechanisms being more than adequate for regression benchmarks. The automated data acquisition must be able to recognize when the regression benchmark outputs data that describe the regular behavior of the middleware as opposed to data distorted by the warm up period at the start of the benchmark. Middleware benchmarking in general either uses long warm up periods or expects the warm up periods to be set by trial and error, neither of which is acceptable for regression benchmarks. The automated results analysis remains a significant obstacle, as it must be able to detect performance regressions quickly and reliably. The longer the period between the occurrence and detection of a performance regression, the more difficult it is to find the source of the regression and the more costly it is to fix it. This requirement implies a need for benchmarks that are so short they can be run daily and so precise they can detect minuscule changes in performance. Detection of a performance regression is especially a problem for creeping performance degradation, which consists of a sequence of individually negligible changes over a long period of time. Middleware benchmarking in general relies on a manual results analysis, which is not feasible for the large quantities of results produced by regular runs of regression benchmarks. 3. Simple Regression Benchmarks The existing middleware benchmarks in general can be divided into two broad groups based on their complexity. The group of relatively simple benchmarks covers benchmarks such as [4][5][6], where an isolated feature of the middleware is tested under artificial workload. The intuitive justification for the simple benchmarks is that they provide precise results that have a straightforward interpretation. The group of relatively complex benchmarks covers benchmarks such as [8][10][12], where a set of features of the middleware is tested under real-world workload. The intuitive justification for the complex benchmarks is that they provide overall results that have a direct relationship to real-world applications. The complex benchmarks do not lend themselves to the regression benchmarking as readily as the simple benchmarks, because they are more expensive to run and because their results have a less straightforward interpretation. In this paper, we focus on the simple benchmarks. Figure 1 Results of consecutive executions of the remote method invocation benchmark (benchmark A) Existing Simple Benchmarks A representative example of the simple middleware benchmarks is a benchmark that measures the duration of a remote method invocation by averaging the dura-
3 tions of several remote method invocations. The results of such a benchmark, similar to the results from [6], are on figure 1. The results on figure 1 have a relative variation of 3.8%. Similar results in [6] suggest that consecutive executions of the same simple benchmark will typically yield results with a relative variation of at least units of percents. Using a simple comparison of the results for regression benchmarking would imply a need to ignore the differences of units of percents and only identify differences of tens of percents as performance regressions. For the simple benchmarks that is clearly too low a resolution Minimizing Result Distortion The difference in the results of consecutive executions of the same simple benchmark can be partially attributed to interference from the operating system, consisting especially of involuntary context switches and device interrupts. Figure 2 shows the interference on a benchmark that measures the duration of a remote method invocation by marking those observations that were interrupted by involuntary context switches and device interrupts. The results were obtained by modifying the operating system to provide the necessary information. the exceptional observations as a method of minimizing the result distortion. An alternative method of minimizing the result distortion is keeping the measured operation duration below the period of the interference and thus making the chance of interference happening during the measured operation reasonably small. We can then express the results using a median rather than an average of the observations, as it is a more robust estimator that is not affected by a small number of exceptional observations. For the remote method invocation, keeping the measured operation duration below the period of the interference means measuring the low-level operations that make up the remote method invocation, such as the marshaling and unmarshaling operations, data conversion operations and dispatching in various stages of the invocation, rather than the entire remote method invocation. The duration of the low-level operations ranges from tens to hundreds of microseconds, which is well below the period of the operating system interference, ranging from tens to hundreds of milliseconds. Assuming the remote method invocation is made up of n similar low-level operations, the duration of the low-level operations will have a relative variation roughly square-root-of-n-times higher than the relative variation of the remote method invocation. This does not imply a decrease in the resolution of the regression benchmark in terms of section 3.1 though, for it is the duration of the remote method invocation rather than the duration of the low-level operations that the resolution should be related to Collecting Enough Observations The reliability of a simple benchmark that reports a result calculated from several observations depends on the number of observations. When estimating the median of the operation duration, we can assume the observed durations to be independent identically distributed observations and then estimate the median using order statistics. Figure 3 shows the relative precision of the median depending on the number of observations, based on an estimate of the confidence interval of the median at the 99% confidence level. Figure 2 Interrupted observations of the remote method invocation benchmark (benchmark A). The results on figure 2 only attribute some of the exceptional observations to the involuntary context switches and device interrupts. The fact that the operating system does not make the information about such interference readily available disqualifies filtering of Alternatively, we can determine the minimal number of observations necessary to ensure a precise estimate of the median using the quantile precision requirement proposed by Chen and Kelton in [2]. The requirement uses a dimensionless maximum proportion confidence half-width instead of the usual maximum absolute or relative confidence half-width. The required number of
4 observations n p for the fixed-sample-size procedure of estimating the p quantile of an independent identically distributed sequence is: ( ε ) 2 ( 1 ) 2 z α p p 1 2 n p 4 provide about 25 times better resolution in terms of section 3.1 than the results on figure 1. On the other hand, the high relative variation makes the results on figure 4 even more unsuitable for a simple comparison than the results on figure 1. where z 2 1-α/2 is the (1-α/2) quantile of the normal distribution, ε is the maximum proportion half-width of the confidence interval, and (1-α) is the confidence level. For a 95% confidence that the median estimator has no more than ε = deviation from the true but unknown median, we obtain a result of n p A choice of n p = borders with 99% confidence level and is acceptable for a simple benchmark. Figure 4 Median results of consecutive executions of the marshaling benchmark (benchmark B). Figure 3 Relative precision of the median for the marshaling benchmark (benchmark B) Still Different Results After minimizing the interference and collecting the necessary number of observations, the consecutive executions of the same simple benchmark will still yield different results. Figure 4 shows the differences on a benchmark that measures the duration of a low-level marshaling operation of a remote method invocation. The results on figure 4 have a relative variation of 8.7% with an average of 3.4 µs, compared to the relative variation of 3.8% with an average of 206 µs on figure 1. Keeping in mind that it is the duration of the remote method invocation rather than the duration of the low-level marshaling operation that the regression benchmarking ultimately monitors, the results on figure Compared to the differences in the results of consecutive executions presented on figure 1, the differences in the results on figure 4 are less due to the results being distorted and unreliable, and more due to the benchmark not having enough control over the initial state of the system to make the results repeatable across executions. For complex benchmarks, things such as the physical placement of files in a filesystem or records in a database can impact the results. Simple benchmarks deal with smaller measured operation durations, therefore even things such as the selection of physical memory pages coupled with limited memory cache associativity can become an issue. Note that this makes the accepted practice of comparing results that are averaged observations at least questionable. To compare the results correctly, it is necessary to treat each result of a benchmark execution as an observation of a random variable. We can assume the results of several consecutive benchmark executions to be a set of independent identically distributed observations with normal distribution. Under this assumption, the sets of results from multiple benchmark executions can be compared using the standard statistical tests for comparing samples from two or more normal populations. We use the two-sample variation f-test to validate the equal variance assumption, and the unmatched twosample t-test to compare the averages of a pair of benchmark result sets.
5 Figure 5 Results of the marshaling benchmark (benchmark B) for consecutive build versions. The results of the technique applied to a real-world example are illustrated on figure 5. The example evaluates the development progress of the marshaling mechanism in TAO for a range of build versions dated from May 19, 2003 to October 27, 2003 with a step of one week. The technique was used to compare the sets of results from multiple benchmark executions for pairs of consecutive build versions. Bold lines mark the two performance changes that were detected in the development process. Note that it would be impossible to detect performance changes by comparing the results of the individual executions. 4. Conclusion Regression benchmarking is a variant of regression testing focused at detecting performance regressions. We introduce regression benchmarking in the area of middleware benchmarking, explaining how the middleware regression benchmarking differs from middleware benchmarking in general. Selecting the broad group of relatively simple benchmarks, we illustrate why the existing benchmarks do not give results sufficient for regression benchmarking. As the next step, we present a set of guidelines on minimizing result distortion and collecting enough observations, and propose a technique for detecting performance regressions using simple benchmarks that adhere to the presented guidelines. Importantly, the technique differs from the accepted practice of comparing results that are averaged observations, which is generally incorrect. We demonstrate this on a realworld example of middleware benchmarking. Although the guidelines and techniques in the paper are sufficient to conduct regression benchmarking with simple middleware benchmarks, further work is necessary to achieve the same results for relatively complex benchmark. Complex middleware benchmarks are indispensable because they exercise multiple functions of the middleware concurrently and therefore measure effects of complex interactions among the functions. Unfortunately, the complex benchmarks are more expensive to run than the simple benchmarks, and their results have a less straightforward interpretation, especially when expressed as a simple value of throughput in a number of operations per second as in [8][10][12]. We are currently investigating the use of clustering [9] to separate the results of complex benchmarks into groups of values that lend themselves better to the automated analysis. Our work on regression benchmarking is available at The current status includes a number of benchmarks and a limited support for automated execution and data acquisition, the support for automated results analysis is being added as the required techniques are developed.
6 5. Acknowledgements The authors would like to remember Adam Buble for his invaluable contribution to our efforts, and to thank Franklin Webber for his insightful review of this paper. This work is partially sponsored by the Grant Agency of the Czech Republic grant 102/03/ References [1] CAROL: Common Architecture for RMI ObjectWeb Layer, [2] E. J. Chen, W. D. Kelton: Simulation-based Estimation of Quantiles, Winter Simulation Conference 99, USA, [3] The Community OpenORB Project, [4] Distributed Systems Research Group: Open CORBA Benchmarking Project, [5] Distributed Systems Research Group: Vendor CORBA Benchmarking Project, [6] DOC Group: TAO Performance Scoreboard, [7] DOC Group: The ACE Orb, [8] ECperf Specification, Version 1.1, Sun Microsystems, 2002, [9] V. Faber, Clustering and Continuous k-means Algorithm, Los Alamos Science, No. 22, [10] ObjectWeb Consortium: RUBiS: Rice University Bidding System, [11] F. Plášil, P. Tůma, A. Buble: Charles University Response to the Benchmark RFI, OMG bench/ , [12] Transaction Processing Performance Council: TPC Benchmark Web Commerce Specification 1.8, 2002, [13] P. Tůma, A. Buble: Open CORBA Benchmarking, SPECTS 01, USA, 2001.
ENVIRONMENTAL REINFORCEMENT LEARNING: A Real-time Learning Architecture for Primitive Behavior Refinement
ENVIRONMENTAL REINFORCEMENT LEARNING: A Real-time Learning Architecture for Primitive Behavior Refinement TaeHoon Anthony Choi, Eunbin Augustine Yim, and Keith L. Doty Machine Intelligence Laboratory Department
More informationCHAPTER 4 CONTENT LECTURE 1 November :28 AM
CHAPTER 4 By Radu Muresan University of Guelph Page 1 CHAPTER 4 CONTENT LECTURE 1 November 07 12 10:28 AM UNIPROCESSOR SCHEDULING Real Time Task Model Concepts Types of Real Time Tasks and Their Characteristics
More informationTrace-Context Sensitive Performance Profiling for Enterprise Software Applications
Trace-Context Sensitive Performance Profiling for Enterprise Software Applications Presentation at SPEC International Performance Evaluation Workshop 2008 Matthias Rohr 1, André van Hoorn 1, Simon Giesecke
More informationNEW METHODS FOR SENSITIVITY TESTS OF EXPLOSIVE DEVICES
NEW METHODS FOR SENSITIVITY TESTS OF EXPLOSIVE DEVICES Amit Teller 1, David M. Steinberg 2, Lina Teper 1, Rotem Rozenblum 2, Liran Mendel 2, and Mordechai Jaeger 2 1 RAFAEL, POB 2250, Haifa, 3102102, Israel
More informationStatistical analysis DIANA SAPLACAN 2017 * SLIDES ADAPTED BASED ON LECTURE NOTES BY ALMA LEORA CULEN
Statistical analysis DIANA SAPLACAN 2017 * SLIDES ADAPTED BASED ON LECTURE NOTES BY ALMA LEORA CULEN Vs. 2 Background 3 There are different types of research methods to study behaviour: Descriptive: observations,
More informationMODELING AN SMT LINE TO IMPROVE THROUGHPUT
As originally published in the SMTA Proceedings MODELING AN SMT LINE TO IMPROVE THROUGHPUT Gregory Vance Rockwell Automation, Inc. Mayfield Heights, OH, USA gjvance@ra.rockwell.com Todd Vick Universal
More informationA STATISTICAL PATTERN RECOGNITION PARADIGM FOR VIBRATION-BASED STRUCTURAL HEALTH MONITORING
A STATISTICAL PATTERN RECOGNITION PARADIGM FOR VIBRATION-BASED STRUCTURAL HEALTH MONITORING HOON SOHN Postdoctoral Research Fellow ESA-EA, MS C96 Los Alamos National Laboratory Los Alamos, NM 87545 CHARLES
More informationTEACHING REGRESSION WITH SIMULATION. John H. Walker. Statistics Department California Polytechnic State University San Luis Obispo, CA 93407, U.S.A.
Proceedings of the 004 Winter Simulation Conference R G Ingalls, M D Rossetti, J S Smith, and B A Peters, eds TEACHING REGRESSION WITH SIMULATION John H Walker Statistics Department California Polytechnic
More informationComment on McLeod and Hume, Overlapping Mental Operations in Serial Performance with Preview: Typing
THE QUARTERLY JOURNAL OF EXPERIMENTAL PSYCHOLOGY, 1994, 47A (1) 201-205 Comment on McLeod and Hume, Overlapping Mental Operations in Serial Performance with Preview: Typing Harold Pashler University of
More informationCorrelation vs. Causation - and What Are the Implications for Our Project? By Michael Reames and Gabriel Kemeny
Correlation vs. Causation - and What Are the Implications for Our Project? By Michael Reames and Gabriel Kemeny In problem solving, accurately establishing and validating root causes are vital to improving
More informationComparison of volume estimation methods for pancreatic islet cells
Comparison of volume estimation methods for pancreatic islet cells Jiří Dvořák a,b, Jan Švihlíkb,c, David Habart d, and Jan Kybic b a Department of Probability and Mathematical Statistics, Faculty of Mathematics
More informationA NEW, ADVANCED HIGH- THROUGHPUT SYSTEM FOR AUTOMATED INHALER TESTING
A NEW, ADVANCED HIGH- THROUGHPUT SYSTEM FOR AUTOMATED INHALER TESTING Two years ago, Novi Systems Ltd set out to shake up the inhaler automation market. On December 8th, 2015, at the Drug Delivery to the
More informationADVANCED TECHNIQUES FOR THE VERIFICATION AND VALIDATION OF PROGNOSTICS & HEALTH MANAGEMENT CAPABILITIES
ADVANCED TECHNIQUES FOR THE VERIFICATION AND VALIDATION OF PROGNOSTICS & HEALTH MANAGEMENT CAPABILITIES James E. Dzakowic G. Scott Valentine Impact Technologies, LLC 00 Canal View Blvd Rochester, NY 463
More informationJ2.6 Imputation of missing data with nonlinear relationships
Sixth Conference on Artificial Intelligence Applications to Environmental Science 88th AMS Annual Meeting, New Orleans, LA 20-24 January 2008 J2.6 Imputation of missing with nonlinear relationships Michael
More informationData and Statistics 101: Key Concepts in the Collection, Analysis, and Application of Child Welfare Data
TECHNICAL REPORT Data and Statistics 101: Key Concepts in the Collection, Analysis, and Application of Child Welfare Data CONTENTS Executive Summary...1 Introduction...2 Overview of Data Analysis Concepts...2
More informationIS SOURCE-CODE ISOLATION VIABLE FOR PERFORMANCE CHARACTERIZATION?
IS SOURCE-CODE ISOLATION VIABLE FOR PERFORMANCE CHARACTERIZATION? C. Akel, Y. Kashnikov, P. de Oliveira Castro, W. Jalby University of Versailles Exascale Computing Research 2 Why extracting code? Problem
More informationA Brief Introduction to Bayesian Statistics
A Brief Introduction to Statistics David Kaplan Department of Educational Psychology Methods for Social Policy Research and, Washington, DC 2017 1 / 37 The Reverend Thomas Bayes, 1701 1761 2 / 37 Pierre-Simon
More informationSelf-aware Early Warning Score System for IoT-Based Personalized Healthcare
Self-aware Early Warning Score System for IoT-Based Personalized Healthcare Iman Azimi 1(B), Arman Anzanpour 1, Amir M. Rahmani 1,2, Pasi Liljeberg 1, and Hannu Tenhunen 1,2 1 Department of Information
More informationReliability of feedback fechanism based on root cause defect analysis - case study
Annales UMCS Informatica AI XI, 4 (2011) 21 32 DOI: 10.2478/v10065-011-0037-0 Reliability of feedback fechanism based on root cause defect analysis - case study Marek G. Stochel 1 1 Motorola Solutions
More informationISC- GRADE XI HUMANITIES ( ) PSYCHOLOGY. Chapter 2- Methods of Psychology
ISC- GRADE XI HUMANITIES (2018-19) PSYCHOLOGY Chapter 2- Methods of Psychology OUTLINE OF THE CHAPTER (i) Scientific Methods in Psychology -observation, case study, surveys, psychological tests, experimentation
More informationJohn Quigley, Tim Bedford, Lesley Walls Department of Management Science, University of Strathclyde, Glasgow
Empirical Bayes Estimates of Development Reliability for One Shot Devices John Quigley, Tim Bedford, Lesley Walls Department of Management Science, University of Strathclyde, Glasgow This article describes
More informationAssignment Question Paper I
Subject : - Discrete Mathematics Maximum Marks : 30 1. Define Harmonic Mean (H.M.) of two given numbers relation between A.M.,G.M. &H.M.? 2. How we can represent the set & notation, define types of sets?
More informationFeasibility Evaluation of a Novel Ultrasonic Method for Prosthetic Control ECE-492/3 Senior Design Project Fall 2011
Feasibility Evaluation of a Novel Ultrasonic Method for Prosthetic Control ECE-492/3 Senior Design Project Fall 2011 Electrical and Computer Engineering Department Volgenau School of Engineering George
More informationReinforcement Learning : Theory and Practice - Programming Assignment 1
Reinforcement Learning : Theory and Practice - Programming Assignment 1 August 2016 Background It is well known in Game Theory that the game of Rock, Paper, Scissors has one and only one Nash Equilibrium.
More informationAssignment 4: True or Quasi-Experiment
Assignment 4: True or Quasi-Experiment Objectives: After completing this assignment, you will be able to Evaluate when you must use an experiment to answer a research question Develop statistical hypotheses
More informationComputer Science 101 Project 2: Predator Prey Model
Computer Science 101 Project 2: Predator Prey Model Real-life situations usually are complicated and difficult to model exactly because of the large number of variables present in real systems. Computer
More informationAQC93, 47 th Annual Quality Congress, Boston, Massachusetts, May 24-26, 1993
H. J. Bajaria, Ph.D., P.E. Multiface, Inc. Garden City, Michigan ABSTRACT STATISTICAL PROBLEM SOLVING STRATEGIES Statistical Problem Solving (SPS) strategies play a key role in connecting problem-solving
More informationDifferences of Face and Object Recognition in Utilizing Early Visual Information
Differences of Face and Object Recognition in Utilizing Early Visual Information Peter Kalocsai and Irving Biederman Department of Psychology and Computer Science University of Southern California Los
More informationTC65B, WG6. IEC Industrial Process Control Systems Guideline for evaluating process control systems. Micaela Caserza Magro Paolo Pinceti
COMITATO ELETTROTECNICO ITALIANO TC65B, WG6 IEC 62603 Industrial Process Control s Guideline for evaluating process control systems Micaela Caserza Magro Paolo Pinceti The origin of the idea F UK Group
More informationImpact Evaluation Toolbox
Impact Evaluation Toolbox Gautam Rao University of California, Berkeley * ** Presentation credit: Temina Madon Impact Evaluation 1) The final outcomes we care about - Identify and measure them Measuring
More informationConfidence Intervals On Subsets May Be Misleading
Journal of Modern Applied Statistical Methods Volume 3 Issue 2 Article 2 11-1-2004 Confidence Intervals On Subsets May Be Misleading Juliet Popper Shaffer University of California, Berkeley, shaffer@stat.berkeley.edu
More informationCertification in Structural Health Monitoring Systems
Certification in Structural Health Monitoring Systems C. M. S. KABBAN 1 and M. M. DERRISO 2 ABSTRACT Despite advances in health monitoring systems, the certification process for automated SHM systems remain
More informationDetection of aneuploidy in a single cell using the Ion ReproSeq PGS View Kit
APPLICATION NOTE Ion PGM System Detection of aneuploidy in a single cell using the Ion ReproSeq PGS View Kit Key findings The Ion PGM System, in concert with the Ion ReproSeq PGS View Kit and Ion Reporter
More informationPaper 1 (1827/01 Full Course) - Factors Affecting Participation and Performance
June 2005 GCSE Physical Education Examiners Report Paper 1 (1827/01 Full Course) - Factors Affecting Participation and Performance General Comments As with last year, the vast majority of candidates confined
More informationVarious Approaches to Szroeter s Test for Regression Quantiles
The International Scientific Conference INPROFORUM 2017, November 9, 2017, České Budějovice, 361-365, ISBN 978-80-7394-667-8. Various Approaches to Szroeter s Test for Regression Quantiles Jan Kalina,
More informationUnderstanding Correlations The Powerful Relationship between Two Independent Variables
Understanding Correlations The Powerful Relationship between Two Independent Variables Dr. Robert Tippie, PhD I n this scientific paper we will discuss the significance of the Pearson r Correlation Coefficient
More informationCOMMITTEE FOR PROPRIETARY MEDICINAL PRODUCTS (CPMP) POINTS TO CONSIDER ON MISSING DATA
The European Agency for the Evaluation of Medicinal Products Evaluation of Medicines for Human Use London, 15 November 2001 CPMP/EWP/1776/99 COMMITTEE FOR PROPRIETARY MEDICINAL PRODUCTS (CPMP) POINTS TO
More informationDraft Broadcasting Services (Television Captioning) Standard 2013
Draft Broadcasting Services (Television Captioning) Standard 2013 Submission by the Australian Communications Consumer Action Network to the Australian Communications and Media Authority 22 January 2013
More informationlab exam lab exam Experimental Design Experimental Design when: Nov 27 - Dec 1 format: length = 1 hour each lab section divided in two
lab exam when: Nov 27 - Dec 1 length = 1 hour each lab section divided in two register for the exam in your section so there is a computer reserved for you If you write in the 1st hour, you can t leave
More informationSummarizing Data. (Ch 1.1, 1.3, , 2.4.3, 2.5)
1 Summarizing Data (Ch 1.1, 1.3, 1.10-1.13, 2.4.3, 2.5) Populations and Samples An investigation of some characteristic of a population of interest. Example: You want to study the average GPA of juniors
More informationImproved Intelligent Classification Technique Based On Support Vector Machines
Improved Intelligent Classification Technique Based On Support Vector Machines V.Vani Asst.Professor,Department of Computer Science,JJ College of Arts and Science,Pudukkottai. Abstract:An abnormal growth
More informationAutomated Detection of Performance Regressions Using Regression Models on Clustered Performance Counters
Automated Detection of Performance Regressions Using Regression Models on Clustered Performance Counters Weiyi Shang, Ahmed E. Hassan Software Analysis and Intelligence Lab (SAIL) Queen s University, Kingston,
More informationAbility to link signs/symptoms of current patient to previous clinical encounters; allows filtering of info to produce broad. differential.
Patient Care Novice Advanced Information gathering Organization of responsibilities Transfer of Care Physical Examination Decision Making Development and execution of plans Gathers too much/little info;
More informationEXERCISE: HOW TO DO POWER CALCULATIONS IN OPTIMAL DESIGN SOFTWARE
...... EXERCISE: HOW TO DO POWER CALCULATIONS IN OPTIMAL DESIGN SOFTWARE TABLE OF CONTENTS 73TKey Vocabulary37T... 1 73TIntroduction37T... 73TUsing the Optimal Design Software37T... 73TEstimating Sample
More informationEvolutionary Programming
Evolutionary Programming Searching Problem Spaces William Power April 24, 2016 1 Evolutionary Programming Can we solve problems by mi:micing the evolutionary process? Evolutionary programming is a methodology
More informationResearch Methods in Forest Sciences: Learning Diary. Yoko Lu December Research process
Research Methods in Forest Sciences: Learning Diary Yoko Lu 285122 9 December 2016 1. Research process It is important to pursue and apply knowledge and understand the world under both natural and social
More informationTitle: A robustness study of parametric and non-parametric tests in Model-Based Multifactor Dimensionality Reduction for epistasis detection
Author's response to reviews Title: A robustness study of parametric and non-parametric tests in Model-Based Multifactor Dimensionality Reduction for epistasis detection Authors: Jestinah M Mahachie John
More informationRemarks on Bayesian Control Charts
Remarks on Bayesian Control Charts Amir Ahmadi-Javid * and Mohsen Ebadi Department of Industrial Engineering, Amirkabir University of Technology, Tehran, Iran * Corresponding author; email address: ahmadi_javid@aut.ac.ir
More informationNeuro-MEP-Micro EMG EP. 2-Channel Portable EMG and NCS System with a Built-in Miniature Dedicated Keyboard. EMG according to international standards
Neuro-MEP-Micro 2-Channel Portable EMG and NCS System with a Built-in Miniature Dedicated Keyboard EMG according to international standards Instant analysis of high-quality responses Over 50 EMG and EP
More informationTWO HANDED SIGN LANGUAGE RECOGNITION SYSTEM USING IMAGE PROCESSING
134 TWO HANDED SIGN LANGUAGE RECOGNITION SYSTEM USING IMAGE PROCESSING H.F.S.M.Fonseka 1, J.T.Jonathan 2, P.Sabeshan 3 and M.B.Dissanayaka 4 1 Department of Electrical And Electronic Engineering, Faculty
More informationChapter 1. Introduction
Chapter 1 Introduction 1.1 Motivation and Goals The increasing availability and decreasing cost of high-throughput (HT) technologies coupled with the availability of computational tools and data form a
More informationArt to the aid of technology
Reading Practice Art to the aid of technology What caricatures can teach us about facial recognition A Our brains are incredibly agile machines, and it is hard to think of anything they do more efficiently
More informationComputational Neuroscience. Instructor: Odelia Schwartz
Computational Neuroscience 2017 1 Instructor: Odelia Schwartz From the NIH web site: Committee report: Brain 2025: A Scientific Vision (from 2014) #1. Discovering diversity: Identify and provide experimental
More informationPlacebo and Belief Effects: Optimal Design for Randomized Trials
Placebo and Belief Effects: Optimal Design for Randomized Trials Scott Ogawa & Ken Onishi 2 Department of Economics Northwestern University Abstract The mere possibility of receiving a placebo during a
More informationPythia WEB ENABLED TIMED INFLUENCE NET MODELING TOOL SAL. Lee W. Wagenhals Alexander H. Levis
Pythia WEB ENABLED TIMED INFLUENCE NET MODELING TOOL Lee W. Wagenhals Alexander H. Levis ,@gmu.edu Adversary Behavioral Modeling Maxwell AFB, Montgomery AL March 8-9, 2007 1 Outline Pythia
More informationSparse Coding in Sparse Winner Networks
Sparse Coding in Sparse Winner Networks Janusz A. Starzyk 1, Yinyin Liu 1, David Vogel 2 1 School of Electrical Engineering & Computer Science Ohio University, Athens, OH 45701 {starzyk, yliu}@bobcat.ent.ohiou.edu
More informationThe RoB 2.0 tool (individually randomized, cross-over trials)
The RoB 2.0 tool (individually randomized, cross-over trials) Study design Randomized parallel group trial Cluster-randomized trial Randomized cross-over or other matched design Specify which outcome is
More informationWhere No Interface Has Gone Before: What Can the Phaser Teach Us About Label Usage in HCI?
Where No Interface Has Gone Before: What Can the Phaser Teach Us About Label Usage in HCI? Franklin P. Tamborello, II Phillip H. Chung Michael D. Byrne Rice University Department of Psychology 61 S. Main,
More informationBasic Statistics for Comparing the Centers of Continuous Data From Two Groups
STATS CONSULTANT Basic Statistics for Comparing the Centers of Continuous Data From Two Groups Matt Hall, PhD, Troy Richardson, PhD Comparing continuous data across groups is paramount in research and
More informationArtificial Intelligence AI for Smarter Healthcare
Artificial Intelligence AI for Smarter Healthcare What is Artificial Intelligence? What is Artificial Intelligence? Artificial Intelligence is a way of making a computer, a computer-controlled robot,
More informationSix Sigma Glossary Lean 6 Society
Six Sigma Glossary Lean 6 Society ABSCISSA ACCEPTANCE REGION ALPHA RISK ALTERNATIVE HYPOTHESIS ASSIGNABLE CAUSE ASSIGNABLE VARIATIONS The horizontal axis of a graph The region of values for which the null
More informationLAB ASSIGNMENT 4 INFERENCES FOR NUMERICAL DATA. Comparison of Cancer Survival*
LAB ASSIGNMENT 4 1 INFERENCES FOR NUMERICAL DATA In this lab assignment, you will analyze the data from a study to compare survival times of patients of both genders with different primary cancers. First,
More informationRAG Rating Indicator Values
Technical Guide RAG Rating Indicator Values Introduction This document sets out Public Health England s standard approach to the use of RAG ratings for indicator values in relation to comparator or benchmark
More informationPINPOINTING RADIATION THERAPY WITH THE PRECISION OF MR.
GE Healthcare PINPOINTING RADIATION THERAPY WITH THE PRECISION OF MR. MR Radiation Oncology Suite MAXIMIZE YOUR PRECISION. HELP MINIMIZE PATIENT COMPLICATIONS. Our goal in MR radiation oncology is to
More informationarxiv: v1 [stat.ml] 23 Jan 2017
Learning what to look in chest X-rays with a recurrent visual attention model arxiv:1701.06452v1 [stat.ml] 23 Jan 2017 Petros-Pavlos Ypsilantis Department of Biomedical Engineering King s College London
More informationEconomic Value Lab Efficiencies Safety Clinical Outcomes. Visit BD Booth AACC/ASCLS 2009 Annual Meeting
Economic Value Lab Efficiencies Safety Clinical Outcomes Visit BD Booth 1225 AACC/ASCLS 2009 Annual Meeting BD Events & Co-Promotions Join us this year at booth #1225 to discover the full spectrum of solutions
More informationIntroduction to Experiment Design
Performance Evaluation: Introduction to Experiment Design Hongwei Zhang http://www.cs.wayne.edu/~hzhang The first ninety percent of the task takes ten percent of the time, and the last ten percent takes
More informationMBA SEMESTER III. MB0050 Research Methodology- 4 Credits. (Book ID: B1206 ) Assignment Set- 1 (60 Marks)
MBA SEMESTER III MB0050 Research Methodology- 4 Credits (Book ID: B1206 ) Assignment Set- 1 (60 Marks) Note: Each question carries 10 Marks. Answer all the questions Q1. a. Differentiate between nominal,
More informationMeasuring Focused Attention Using Fixation Inner-Density
Measuring Focused Attention Using Fixation Inner-Density Wen Liu, Mina Shojaeizadeh, Soussan Djamasbi, Andrew C. Trapp User Experience & Decision Making Research Laboratory, Worcester Polytechnic Institute
More informationCognitive Strategies and Eye Movements for Searching Hierarchical Displays
Cognitive Strategies and Eye Movements for Searching Hierarchical Displays Anthony J. Hornof Tim Halverson University of Oregon Sponsored by ONR Three Main Points A hierarchical display motivates specific
More informationA Data Mining Approach for Signal Detection and Analysis
SHORT COMMUNICATION Drug Safety 2002; 25 (6): 393-397 0114-5916/02/0006-0393/$25.00/0 Adis International Limited. All rights reserved. A Data Mining Approach for Signal Detection and Analysis Andrew Bate,
More informationTitle:Mixed-strain Housing for Female C57BL/6, DBA/2, and BALB/c Mice: Validating a Split-plot Design that promotes Refinement and Reduction
Author's response to reviews Title:Mixed-strain Housing for Female C57BL/6, DBA/2, and BALB/c Mice: Validating a Split-plot Design that promotes Refinement and Reduction Authors: Michael Walker Mr (mwalk04@uoguelph.ca)
More informationn Outline final paper, add to outline as research progresses n Update literature review periodically (check citeseer)
Project Dilemmas How do I know when I m done? How do I know what I ve accomplished? clearly define focus/goal from beginning design a search method that handles plateaus improve some ML method s robustness
More informationDay 11: Measures of Association and ANOVA
Day 11: Measures of Association and ANOVA Daniel J. Mallinson School of Public Affairs Penn State Harrisburg mallinson@psu.edu PADM-HADM 503 Mallinson Day 11 November 2, 2017 1 / 45 Road map Measures of
More informationBiology 345: Biometry Fall 2005 SONOMA STATE UNIVERSITY Lab Exercise 8 One Way ANOVA and comparisons among means Introduction
Biology 345: Biometry Fall 2005 SONOMA STATE UNIVERSITY Lab Exercise 8 One Way ANOVA and comparisons among means Introduction In this exercise, we will conduct one-way analyses of variance using two different
More information(Visual) Attention. October 3, PSY Visual Attention 1
(Visual) Attention Perception and awareness of a visual object seems to involve attending to the object. Do we have to attend to an object to perceive it? Some tasks seem to proceed with little or no attention
More informationTo open a CMA file > Download and Save file Start CMA Open file from within CMA
Example name Effect size Analysis type Level Tamiflu Symptom relief Mean difference (Hours to relief) Basic Basic Reference Cochrane Figure 4 Synopsis We have a series of studies that evaluated the effect
More informationDiscrimination and Generalization in Pattern Categorization: A Case for Elemental Associative Learning
Discrimination and Generalization in Pattern Categorization: A Case for Elemental Associative Learning E. J. Livesey (el253@cam.ac.uk) P. J. C. Broadhurst (pjcb3@cam.ac.uk) I. P. L. McLaren (iplm2@cam.ac.uk)
More informationAgent-Based Models. Maksudul Alam, Wei Wang
Agent-Based Models Maksudul Alam, Wei Wang Outline Literature Review about Agent-Based model Modeling disease outbreaks in realistic urban social Networks EpiSimdemics: an Efficient Algorithm for Simulating
More informationSupplementary materials for: Executive control processes underlying multi- item working memory
Supplementary materials for: Executive control processes underlying multi- item working memory Antonio H. Lara & Jonathan D. Wallis Supplementary Figure 1 Supplementary Figure 1. Behavioral measures of
More informationThe Loss of Heterozygosity (LOH) Algorithm in Genotyping Console 2.0
The Loss of Heterozygosity (LOH) Algorithm in Genotyping Console 2.0 Introduction Loss of erozygosity (LOH) represents the loss of allelic differences. The SNP markers on the SNP Array 6.0 can be used
More informationUltrasonic Measuring Stations vs. Mechanical Stadiometers
EVALUATING PATIENT HEIGHT: Ultrasonic Measuring Stations vs. Mechanical Stadiometers The accurate assessment of patient height is one of the key measurements in patient care. Mechanical stadiometers, commonly
More informationRe: ENSC 370 Project Gerbil Functional Specifications
Simon Fraser University Burnaby, BC V5A 1S6 trac-tech@sfu.ca February, 16, 1999 Dr. Andrew Rawicz School of Engineering Science Simon Fraser University Burnaby, BC V5A 1S6 Re: ENSC 370 Project Gerbil Functional
More informationEnhanced Asthma Management with Mobile Communication
Enhanced Asthma Management with Mobile Communication P.S. Ngai, S. Chan, C.T. Lau, K.M. Lau Abstract In this paper, we propose a prototype system to enhance the management of asthma condition in patients
More informationIllumina 3D Portfolio
illumina 3D Portfolio Only the Illumina 3D Portfolio can customize therapy by delivering multiple waveforms to a precise neural target. Illumina 3D Portfolio Right Target. Right Waveform. Customized Therapy.
More informationName of the paper: Effective Development and Testing using TDD. Name of Project Teams: Conversion Team and Patient Access Team.
1 Name of the paper: Effective Development and Testing using TDD Name of the author: Payal Sen E-Mail Id: payal.sen@siemens.com Contact Number: 9748492800 Name of Project Teams: Conversion Team and Patient
More informationBackground Information
Background Information Erlangen, November 26, 2017 RSNA 2017 in Chicago: South Building, Hall A, Booth 1937 Artificial intelligence: Transforming data into knowledge for better care Inspired by neural
More informationContinuous/Discrete Non Parametric Bayesian Belief Nets with UNICORN and UNINET
Continuous/Discrete Non Parametric Bayesian Belief Nets with UNICORN and UNINET R.M. Cooke 1, D. Kurowicka, A. M. Hanea, O. Morales, D. A. Ababei Department of Mathematics Delft University of Technology
More informationEffects of Overweight Samples and Rounding of Grade Percentages on Peanut Grades and Prices
Effects of Overweight Samples and Rounding of Grade Percentages on Peanut Grades and Prices E. F. Pebe Diaz B. W. Brorsen* K. B. Anderson and P. Kenkel Running title (Peanut Grading) 1 Research Assistant,
More informationRecent developments for combining evidence within evidence streams: bias-adjusted meta-analysis
EFSA/EBTC Colloquium, 25 October 2017 Recent developments for combining evidence within evidence streams: bias-adjusted meta-analysis Julian Higgins University of Bristol 1 Introduction to concepts Standard
More informationProbability and Statistics Chapter 1 Notes
Probability and Statistics Chapter 1 Notes I Section 1-1 A is the science of collecting, organizing, analyzing, and interpreting data in order to make decisions 1 is information coming from observations,
More informationUsing Probabilistic Methods to Optimize Data Entry in Accrual of Patients to Clinical Trials
Using Probabilistic Methods to Optimize Data Entry in Accrual of Patients to Clinical Trials Bhavesh D. Goswami, Lawrence O. Hall, Dmitry B. Goldgof, Eugene Fink, and Jeffrey P. Krischer bgoswami@csee.usf.edu,
More informationCourse summary, final remarks
Course "Empirical Evaluation in Informatics" Prof. Dr. Lutz Prechelt Freie Universität Berlin, Institut für Informatik http://www.inf.fu-berlin.de/inst/ag-se/ Role of empiricism Generic method Concrete
More informationIntroduction to Machine Learning. Katherine Heller Deep Learning Summer School 2018
Introduction to Machine Learning Katherine Heller Deep Learning Summer School 2018 Outline Kinds of machine learning Linear regression Regularization Bayesian methods Logistic Regression Why we do this
More informationNot all NLP is Created Equal:
Not all NLP is Created Equal: CAC Technology Underpinnings that Drive Accuracy, Experience and Overall Revenue Performance Page 1 Performance Perspectives Health care financial leaders and health information
More informationDetecting Cognitive States Using Machine Learning
Detecting Cognitive States Using Machine Learning Xuerui Wang & Tom Mitchell Center for Automated Learning and Discovery School of Computer Science Carnegie Mellon University xuerui,tom.mitchell @cs.cmu.edu
More informationHow To Optimize Your Training For i3 Mindware v.4 And Why 2G N-Back Brain Training Works
How To Optimize Your Training For i3 Mindware v.4 And Why 2G N-Back Brain Training Works Mark Ashton Smith, Ph.D. CONTENTS I. KEY SCIENTIFIC BACKGROUND 3 II. PRACTICE: WHAT STRATEGIES? 10 III. SCHEDULING
More informationPROF A.A. OROGADE MBBS, FWACP,
PROPOSAL ON NATIONAL REGISTRY FOR INFECTIVE ENDOCARDITIS IN CHILDREN (PIER) PROF A.A. OROGADE MBBS, FWACP, MSc (Lond), DLSHTM AHMADU BELLO UNIVERSITY ZARIA PROF A.A. OROGADE 2 Outline INFECTIVE ENDOCARDITIS
More informationMEA DISCUSSION PAPERS
Inference Problems under a Special Form of Heteroskedasticity Helmut Farbmacher, Heinrich Kögel 03-2015 MEA DISCUSSION PAPERS mea Amalienstr. 33_D-80799 Munich_Phone+49 89 38602-355_Fax +49 89 38602-390_www.mea.mpisoc.mpg.de
More informationAn ECG Beat Classification Using Adaptive Neuro- Fuzzy Inference System
An ECG Beat Classification Using Adaptive Neuro- Fuzzy Inference System Pramod R. Bokde Department of Electronics Engineering, Priyadarshini Bhagwati College of Engineering, Nagpur, India Abstract Electrocardiography
More information