Improvement on ABDOM-Q d and its Application in Open-Source Community Software Defect Discovery Process

Transcription

1 Improvement on ABDOM-Q d and its Application in Open-Source Community Software Defect Discovery Process Zhitao He School of Computer Science and Engineering, Beihang University , Beijing, China zhitaohe@vip.sina.com Haihua Yan School of Computer Science and Engineering, Beihang University , Beijing, China yhh@buaa.edu.cn Abstract ABDOM-Q d is a model to describe the characteristics of software defect discovery time-ordered process, such as periodicity, attenuation, oscillation, incremental and discrete. It can help testing participants to evaluate testing quality and to predict testing process with the time-ordered software defect discovery ams in a good organized software testing process. Due to the poor organization in open-source software community, software defect discovery data show their obvious uncertainties such as mutations and randomness. In the early study of ABDOM/ABDOM-Q d, this kind of process was excluded from the discussion. So it becomes an issue that whether ABDOM-Q d model can be applied to open-source community software defect discovery process and reveal new natures and characteristics. In order to answer the question, the normalization of b in ABDOM- Q d were discussed firstly under the curves paradigms and theirs actual significances discussion in conditions b τ >0 and b τ < 0. With the discussion results, a software defect discovery cycle stability coefficient B was proposed and the improved model ABDOM-QB d with stronger describing capability was established. Then ABDOM-QB d was applied in a NASA s open-source project, which is a typical representative of open-source projects, to fit the software defect data and a good fitting result was obtained. Finally with the application results the model s applicability and the characteristics of open-source community software defects discovery process were preliminarily discussed. Keywords: Software Defects, Defect Detecting, Open-source Community, ABDOM-Q d I. INTRODUCTION For a long time, a problem plaguing software test managers is how to assess the progress of a software testing. In a software testing process, test managers need to grasp the test progress in order to adjust the test focus, test strategy and test resources at any time, and to avoid a blind test, to improve the efficiency of software testing and to control the test costs, also help to determine whether the testing can be completed on schedule and assess the quality of tested software. Chao Liu School of Computer Science and Engineering, Beihang University , Beijing, China liuchao@buaa.edu.cn In a software development process, the tested software is always experienced with multiple rounds of testing and modification process. Experiences show that, in the process of a test, the uncertain factors, such as, the defect distribution of tested software, the iterative software development and the testing process, the capacity of defect detecting associated with the experience and inputs of testers have a significant impact on the characteristics of defect discovery process, so that the process shows the timeordered characteristics--the periodicity associated with test rounds, the random oscillation caused by a variety of complex factors, as well as the software defects discovery damped attenuation shown with the evolution of a software defects found/repair process, the multimodal oscillation characteristics due to the incremental software development, the discrete characteristics due to the frequent change in testing organizations. These characteristics can be shown more obviously in a relatively standard software unit, integrated and system testing process with a good organization and management. For example the defect am sequential distribution graph shown in Figure 1 shows the characteristics mentioned above, but less obviously and sufficiently in a relatively non-standard software testing process, such as an open-source community software testing process. Defects s Figure 1. A Sequential Distribution Graph of Software Defects Discovery In the field of software engineering, many time-based software defect prediction achievements[1] can be referenced, such as Rayleigh distribution model[2], the exponential distribution model[3] and S-curve distribution

2 model[4], the ABDOM (Accumulative Bi-damped Oscillation Model)[5]/ABDOM-Q d (Accumulative Bidamped Oscillation Model with Quality Coefficient & Discrete)[6] proposed by He Zhitao in 2010/2012, and the ARMA (p,q)(auto Regressive Moving Average)[7] proposed by Wang Tong and others. The characteristics of a standard software defect discovery process: periodicity, random oscillation and attenuation, can be preliminarily described by ABDOM. And ABDOM-Q d is a discrete and standardized modification model on the basis of ABDOM with a software defect discovery process quality coefficient Q. Compared to ABDOM, ABDOM-Q d can better describe the software defect discovery process with discrete features and can better reveal the quality of test process. But in the study of ABDOM/ABDOM-Q d, due to the describing capacity constraint of ABDOM/ABDOM-Q d, it had been stated that ABDOM can only be applied in the organized standard software testing process, while the non-standard and poor organized software testing process was excluded from the earlier discussion, the most typical one of them is opensource community software testing process. Since open-source community software testing process is also of a widely applied software testing process, then it comes to be a series of significant issues that whether opensource community software testing process can be described by ABDOM-Q d? With it, whether the new characteristics of open-source community software testing process can be found? Whether there are some common characteristics between general and open-source community software testing process? Therefore the objective of paper is to study the b in ABDOM-Q d and get an improved model with a stronger describing capacity, then based on the public software defects time-order data in the international opensource community and the improved model to discuss the performance of characteristics(oscillation, periodicity, attenuation, incremental and discrete) in open-source community software defect discovery process, and to analyze the possible new characters in open-source community software defect discovery process and further expand the application scope and describing capacity of ABDOM-Q d. The retrieval in the international software engineering research field shows that the research on defect detecting model aiming at open-source community software defects discovery process is still in a blank, which indicates that the thesis has innovative significances. II. BRIEF INTRODUCTION ON ABDOM/ABDOM-Q D A. Briefs on ABDOM Accumulative Bi-damped Oscillation Model is a mathematical model having ability to describe the timeordered software defect discovery process in certain conditions. Shown in formula (1) below: D(S,t)= (1) The characteristics of software defect discovery process (time-ordered, periodicity, attenuation and oscillation) get a better description in ABDOM: the characters of timeordered, periodicity and oscillation are expressed through sin (t); the attenuation characteristics in the software defect number and found period are expressed by e t. The influence factors generating the oscillation characteristics (the tested software, the development process and testing process) are expressed in ABDOM model: A max, b and τ reflect the factor of testing process, in which, A max denotes the capacity of test team, τ denotes the defect detecting cycle, b denotes the stability of software defect discovery cycle; R i describes the incremental development impact, the code size ratio of new tested version; a describes the decay rate of software defect discovery, revealing the difficulty degree of software defect discovery and the influence of code s quality on testing process, and also reflects the quality of software testing process in a certain degree. Figure 2. Fitting Results of ABDOM to an Actual Project s Data With ABDOM, a fundamental capacity is obtained to describe software defects discovery time-ordered process with the characteristics such as time-ordered, periodicity, attenuation incremental and oscillation. ABDOM s fitting example to the data in a true project is shown in Figure 2. But the weakness of model is that all parameters are fixed in different tested versions, the discrete changes between the tested versions can t be described clearly, and also the parameters a and b haven t typical values with the clear meaning to evaluate the quality of defect discovery process. B. Briefs on ABDOM-Q d With the introduction of a software defect discovery process quality coefficient Q in ABDOM, and the modification to the value range of R i, and the discretization improvement on A max, Q, b and τ, ABDOM-Q d is an improvement based on ABDOM with discrete and standardized parameters. ABDOM-Q d is shown in formula (2) below: D(S, t) =,

3 = (2) i =1,2,3 k, t i Z +, Z +, τ i Z +, R i (0,1] and =1. Qi (-,5], b i (-, ), Figure 3. Fitting Results of ABDOM-Q d to an Actual Project s Data The most important improvement in ABDOM-Q d is the introducing of coefficient Q, it can be used as an intuitive basis for the evaluation to the quality of testing process with 5 levels: Out of control, Poor, Good, Very good and Excellent. Compared with ABDOM, ABDOM-Q d has the capacity to describe the characteristics in a practical testing process, such as a complex incremental development, the discretization of tested versions, the discrete changing in the test team and the quality of testing process. An example of ABDOM-Q d s fitting to a true project s data is shown in Figure 3, ABDOM-Q d has a stronger describing capacity than ABDOM in describing a discrete and incremental test process, and also the fitting value of parameter Q(0.5 to 0.8) reveals that the quality of discovery process in that example is of middle-poor, a valued evaluation to the process. III. NORMALIZATION OF B IN ABDOM-QD Figure 4 (see the section after the References) shows the true defect discovery data from a NASA public open-source project. Comparing with the curves from standard and organized defect discovery process, the discovery curve in Figure 4, which is a typical software defect discovery curve in open-source community software testing processes, shows its random character obviously. It is clear that describing the defect discovery process in open-source community is really a challenge to ABDOM-Q d. Thus in the thesis, firstly the relationship between the b and τ will be discussed with ABDOM-Q d, with the results of discussion ABDOM-Q d will be modified to have a stronger capacity to describe open-source community software defect discovery process. The normalized parameter Q has been achieved from ABDOM-Q d under the discussion on the relationship of a andτ, by using the Q, the quality of software testing process can be evaluated in a certain degree. The b in the ADBOM- Q d is designed for describing the attenuation amplitude of software defect discovery cycle. The value range of b is definite as, b(-, ). When b=0, indicates that the software defect discovery cycle remains stable; When b>0, indicates that the new software defect discovery peak is to be postponed or disappearing, revealing that the new software defects are increasingly difficult to be found; When b<0, indicates that the new software defect discovery peak comes constantly in advance or is merged with the previous peak, revealing that there may be more defects to be found. Preliminary conclusions can be drawn here that the parameter b describes the density of software defect discovery peaks in tested versions. But the practical meaning of b is still not clear enough since no typical value can be achieved from the direct using of b. From the experience of normalization process of a in ABDOM-Q d, maybe b and τ also perform similarly. A. Discussion on relationship of b τ as b>0 and b<0 Input the following parameters into ABDOM-Q d,: A max =100, R 1 =0.2, R 2 =0.4, R 3 =0.2, R 4 =0.2, lets the software defect discovery curve to be shown in form of a 4 peaks oscillation. In the case of different values of Q, under τ = 10, τ = 20, τ = 50, the variation characteristics of curves are discussed and recorded respectively. Generally the curves show in certain diagrams as b τ gets certain typical values. The diagrams are shown in Figure 5. Figure 5(a). b τ< The defect discovery cycles are sharply shortened; the defects peaks with multi-sub-peaks come frequently Figure 5(b). b τ The defect discovery cycles are slightly shortened; software defect discovery cycles are shorten so that in test versions a new peak is added

4 Figure 5(c). b τ= 0.00 The defect discovery cycles remain stable Figure 5(h). b τ 4.50 No obvious defect discovery cycle Figure 5(d). b τ 0.10 The defect discovery cycles are slightly prolonged so that the subsequent peaks in a tested version overlap with the first peak of next test version Figure 5(e). b τ 0.45 The defect discovery cycles are double prolonged; the subsequent peaks in a tested version disappear completely Figure 5(f). b τ 0.67 The defect discovery cycles are double prolonged with the curves after turning points uplift to their tops Figure 5(g). b τ 1.50 The defect discovery cycles are double prolonged with the maximum values of peaks reduced 50% B. Normalization of b With the above discussion on b τ, let B= b τ, then Definition 1 is given below. Definition 1 Software defect discovery cycle stability coefficient B is a parameter in ABDOM-Q d which can indicate the stability of software defect discovery cycle, B [-, 4.5], the typical values of B and their meanings are listed as follows: B 4.50 No Obvious cycle B 1.50 Double Prolonged /50%Off B 0.67 Double Prolonged /Top B 0.45 Double Prolonged B 0.10 Slightly Prolonged B= 0 Stable B Slightly Shortened B Sharply Shortened B> 0 indicates that software defects are more and more difficultly to be found in a testing process, it is a good signal; B< 0 indicates that software defects are more and more easily to be found, it is a bad signal and the attention is needed as soon as possible. C. Improvement on ABDOM-Q d based on B With Definition 1, ABDOM-QB d is obtained in the formula (3): D(S, t) =, = + (3) i =1,2,3 k, t i Z +, Z +, N, Z +, Q i (-, 5], B i (-, 4.5], R i (0, 1] and =1. Thus, the normalization and discretization on ABDOM has been completed with the clear meanings and typical values of A, a, b, R. And also a new additional parameter A offset can be used to describe the whole deviation of a software defect discovery data. For example, in the left curve in Figure 6, when the software defect discovery time-ordered data shift downward, there exist some vacant sections between the software defect discovery peaks. The describing capacity of ABDOM-QB d has been further improved so that it can describe different types of software defect discovery

5 processes (including different regulation and organization levels). Figure 6 Illustration of A Software Defect Discovery Time-ordered Curve Shifts Downward 2/Upward 2 IV. FITTING TO THE PUBLIC DATA IN OPEN-SOURCE COMMUNITY WITH ABDOM-QBD A. Briefs on NASA s Open-source Project PC4 The data in NASA/PC4 was derived from the NASA s Metrics Data Program. PC4 was the flight control software of a geosynchronous satellite in NASA, containing 36,000 lines of C code. The testing period of PC4 was from 1997 to The original public data of NASA/PC4 comes from The data was preprocessed and the 342 test days data acquired from 1997 to 2002 is shown in Figure 4. Obviously the curve has such characteristics as incremental, oscillation, attenuation, discrete and cyclical. The fitting to this curve will verify the validation of ABDOM-QB d. The details of data are shown in TABLE 1. TABLE 1. Details of Software Defect Discovery Data: NASA/PC B. Fitting and Analysis to the Data of NASA/PC4 with ABDOM-QB d Eventually an accurate fitting result was gotten with ABDOM-QB d, the fitting error was 1.86, the values of various parameters show some clear and stable characteristics. The fitting results are satisfying and ABDOM-QB d shows its stronger capacity to describe opensource community software defect discovery process. The fitting results is shown in Figure 7(see the section after the References), the fitting data of various parameters are shown in Table 2. With the data in Table 2, it can be drawn the following analysis: 1) Open-source community software defect discovery process is essentially a discrete incremental software testing process, its periodicity, oscillation, attenuation, incremental and discrete characters are shown apparently. The data show that there exist 38 software defect discovery peaks in NASA/PC4 project, A max =430, a steady value shown, it reveals that the testers participating in NASA/PC4 were about persons. Q i, B i and τ i presents random vibration trend within a certain range. In overall, Q, B and τ tends to be a stable value, the average value of Q is 0.94, the average value of B is 0.087, and the average value of τ is ong them, the lower value of Q indicates that the quality of software defect discovery process is poor. TABLE 2. Fitting Data to NASA/PC4 with ABDOM-QB d The Mean-square Deviation of Fitting Error is R =0.997, A offset = 1 Start Points of Peaks A max Q i B i τ i R i

6 The Mean-square Deviation of Fitting Error is R =0.997, A offset = 1 Start Points of Peaks A max Q i B i τ i R i ) Open-source community software defect discovery process has obvious characteristic of randomness. Judging from the specific data, the values of Q, B and τ oscillating in an unknown rule is shown in Figure 8. The randomness character shown in open-source community software defect discovery process obviously comes from the loose and casual organization and management in the open-source community testing process. Compared with the certainty and closeness of standard software testing process, opensource community software testing process shows its openness and uncertainty. values of B is approximately limited in 0.15~0.2. The curve paradigm is shown in Figure 9, it is found that in the end section of NASA/PC4 s fitting curve, such a paradigm curve appears 4 times and totally appears 13 times in the whole process. This is likely to be a signal of the software defect discovery process being in a convergence phase. Although this open-source community software testing was in loose organization, but the common goal to achieve a quality product in the late period of project played a strong role in the supervision and management to the loose team of the open-source community. Figure 9. The curve paradigm as Q=1.5, B=0.2 V. CONCLUSIONS With the study on software defect cycle stability, a software defects discovery cycle stability coefficient B and a new ABDOM-QB d are defined in the article. With ABDOM-QB d, the public defect discovery data in NASA/PC4 (a typical representative of open-source community software testing process) were fitted and the fitting results can better describe the characteristics of opensource community software defect discovery process: periodicity, oscillation, attenuation, incremental, discrete and randomness. The results shows that the software defect discovery process in open-source community and in ordinary process are the same in essence. Open-source community software defect discovery process can t be described with ABDOM with the conclusions in former papers, but now it can be described with the modified ABDOM-QB d, the timeordered random oscillation character of Q, B and τ is the hallmark of open-source community software defect discovery process. The subsequent research can be focused on the further analysis of the random oscillation character of Q, B and τ aiming at open-source community software defect discovery process. REFERENCES Figure 8. Time-ordered Distribution of Q and B in ABDOM-QB d fitting results of NASA/PC4 3) Q and B in open-source community software defect discovery process are shown as a significant value combination. In the late period of project NASA/PC4, the values of Q is approximately limited in 1~1.5, and the [1] Wang Qing, Wu Shujian, Li Mingshu. Software Defect Prediction. Journal of Sofware,2008,19(7): (in Chinese) [2] Trachtenberg M. Discovering how to ensure software reliability. RCA Engineer, 1982, 27(1):53-57 [3] Jelinski Z, Moranda P. Software reliability research. In: Freiberger W, ed. Statistical Computer Performance Evaluation. New York: Academic Press, [4] Yamada S, OhbaM, Osaki S. S-Shaped reliability growth modeling for software error detection. IEEE Trans. On Relibility, 1983, R-32(5):

7 [5] He Zhitao, Yan Haihua, Liu Chao. Accumulative Bi-Damped Oscilla tion Model for the Sequential Process of Software Defect Discovery. Journ al of Sofware, 2010,21(12): (in Chinese) [6] He Zhitao, Yan Haihua, Liu Chao. Improvement on ABDOM with n ormalized and discrete parameters. Journal of Beijing University of Aerona utics and Astronautics, 2012, 38(10): (in Chinese) [7] Wang Tong, Wang Liang. Analys is and Forecasting of Bugs Based o n Time Series Model. Modern Computer, Figure 4. Software Defect Discovery Data Time-ordered Distribution: NASA/PC4 Figure 7. Fitting Result Curve of ABDOM-QB d with NASA/PC4 s Data