TDD HQ : Achieving Higher Quality Testing in Test Driven Development

Transcription

1 TDD HQ : Achieving Higher Quality Testing in Test Driven Development Adnan Čaušević, Sasikumar Punnekkat and Daniel Sundmark School of Innovation, Design and Engineering Mälardalen University, Västerås, Sweden {adnan.causevic, sasikumar.punnekkat, daniel.sundmark}@mdh.se Abstract Test driven development (TDD) appears not to be immune to positive test bias effects, as we observed in several empirical studies. In these studies, developers created a significantly larger set of positive tests, but at the same time the number of defects detected with negative tests is significantly higher than those detected by positive ones. In this paper we propose the concept of TDD HQ which is aimed at achieving higher quality of testing in TDD by augmenting the standard TDD with suitable test design techniques. To exemplify this concept, we present combining equivalence partitioning test design technique together with the TDD, for the purpose of improving design of test cases. Initial evaluation of this approach was performed in an academic experiment, where the group who followed this approach created 5% less test cases but at the same time detected 17% more defects compared to a group of students using regular TDD. Keywords Test Driven Development, Test Case Design, Test Efficiency, Negative Testing I. INTRODUCTION Agile methodologies provide us with the new perspective on how we should focus on the quality of software products. In comparison with traditional approaches, testing in agile is not generally perceived as a standalone activity performed by a dedicated quality assurance team somewhere along the very end of the project timeline. Instead, in agile, quality is everyone s business and it is built into the product from the very beginning of the development process. One of the most prominent, quality related, practices in agile is test driven development. Test driven development (TDD) gained on popularity with the introduction of the extreme Programming (XP) methodology [1]. As its name reveals, TDD is a development practice which essentially requires from the developers to first create a test case before writing the code for a given functionality. What is important to note here is that developers now have the full responsibility for creating executable test cases with an automated verdict for the unit or component level of the system that they are developing. A more detailed elaboration, on what is considered as TDD and what exactly developers have to do in it, is presented in section III. TDD is under constant empirical evaluation by practitioners and academics to assess the claimed benefits of code quality improvements following this approach. The current body of knowledge indeed does contain several evidence reports where higher quality of code was produced by simply following TDD instead of more traditional test last approach of software development [2] [6]. However, industrial acceptance of TDD is yet to be fully-fledged. In our industrial survey [7] participants claimed they would prefer very much using TDD in the future, but currently this is not something they do to a full extent. The focus of our research is on identifying particular challenges towards a wide industrial acceptance of TDD and proposing new techniques intended to address those challenges and help TDD gain wider industrial adoption. In section II we present background information and findings of the research we conducted in this area. Section III details proposed improvements in the process of performing TDD with the focus on improving design of the produced test cases. Empirical evaluation of one particular proposed improvement is presented in section IV, while section V elaborate on the related work in this area. Section VI lists conclusions and directions for future work. II. RESEARCH BACKGROUND Once we identified TDD as a preferred, but not widely praticed activity in industry [7], our next goal was to identify and list potential bottlenecks hindering its full industrial acceptance. This was done in the form of a systematic literature review [8], cataloguing the current body of knowledge with respect to empirical studies performed for the purpose of evaluating TDD. We identified seven potentially limiting factors, one of them being developers inability to write efficient and effective test cases. To gain better understanding of this problem and validate significance of such a limiting factor, we performed an empirical study as part of a master-level course on Software Verification & Validation. The main goal of this study [9] was to investigate how developers can benefit from additional knowledge on software test design techniques while performing TDD. Although, we could not establish any statistically significant difference between students performance before and after the course, we did notice a surprisingly high ratio of positive test cases within their test suites. This effect is known as positive test bias [10], [11] where testing is done using more positive or specification defined inputs. However, such a result may not come as a surprise considering that TDD is designed in way to lead developers in the writing of positive tests which will guide them forward in the implementation of a correct functionality. What was interesting to investigate further was that consequences on the quality of testing in TDD we might face if developers focus is only on creating positive test cases.

2 A follow-up study [12] was designed to further investigate if there are any differences in the defect detecting effectiveness of positive and negative test cases created when following the TDD approach. By the term negative test case, we refer to a test case that was created for the purpose of exercising a program in a way that was not explicitly specified in the requirement. On the other hand, a positive test case exercises a program behaviour as it is specified in the requirement. The results of our study again point out a significantly higher number of positive tests compared to negative ones, but at the same time a much higher defects detecting effectiveness of negative tests compared to positive ones. Since this study was conducted in an academic environment, using students as subjects, this result could be interpreted due to the lack of experience of experiment participants. Maybe if our participants were professional developers with many years of experience, their tests would focus on negative aspects to a higher extent and thus limit positive test bias effect. We replicated our academic experiment in an industrial setting, using professional developers as subjects of study, with the intention of confirming existence of a positive test bias effect and more importantly to calculate the defect detecting effectiveness of negative test cases. The results of this study [13] showed a statistically significant significantly...difference between the number of positive and negative test cases. This way we could identify that positive test bias exists even when professional and experienced developers are following TDD. The difference in defect detecting effectiveness, between positive and negative test cases, was also statistically significant. Around 29% of all test cases created by developers were negative, but at the same time contributing in revealing as much as 71% of all the defects found by all test cases. Since we assumed that a positive test bias effect will be identified in this experiment as well, we added one more observation group to our experiment. In addition to dividing experiment participants into test-last (TL) and test driven development (TDD) groups, we also randomly assigned participants to a new group called test-driven development with the support for negative testing group, or short TDD+ group. Our goal with TDD+ group was to investigate if participants would focus more on negative testing by purely informing them that this is needed. We instructed this group of developers to follow TDD, but also to occasionally write a negative test case instead of a positive one, which would be based on the input space, domain knowledge, etc. (but not explicitly stated in the requirements). There was no obvious difference nor improvements in the number of negative test cases created by TDD+ group compared to other two groups. Defect detecting effectiveness did not change as well. We reason that this observation could again be due to the effect of positive thinking that TDD requires from developers. After further analysis of the solutions provided by the participants and the survey questions they answered, we identified a general problem of when to consider convenient to write a negative test case during the test driven development process. We noticed that a more systematic approach in modifying TDD is needed and our idea was to investigate the possibility of extending test-driven development with a particular test design technique. This way we can facilitate consideration of unspecified requirements during the development to a higher extent and thus minimise the impact of a potentially inherent effect of the positive test bias in TDD. Fig. 1. III. Code Refactoring Stop True Passed Failed Start All Features Implemented False Select a new Feature Add a new Test Case Execute all Test Cases Failed Make minimal code changes Execute all Test Cases Test Driven Development Process Flow Passed TDD HQ - HIGHER QUALITY TESTING IN TDD The Test Driven Development process flow is detailed in Figure 1. Although, in simple words, TDD is about writing test case before writing the code, there are few noteworthy steps in the process that should be mentioned here. Execution of test cases happens twice and this at first might seems like a redundant task. However, it is needed for a developer to see that the newly added test case is failing (usually displayed in red) in order to continue working on creating minimal changes to the code to pass that test (this time displayed in green) [14]. In addition, after fully implementing a set of features, developer have to perform refactoring of the code. Refactoring is a process of modifying the code for the purpose of improving its design, readability, maintainability, etc. without violating existing functionality. Because of these mentioned aspects, TDD is sometimes refereed as a red-green-refactor process.

3 True Start All Features Implemented testing effort. It is known that benefits, mostly in terms of cost, are significant when defects are detected at the early stage of development [15]. However, as we discussed in the previous section, test cases created in TDD are not designed to detect defects but rather to provide confidence in the correctness of implemented functionality. Code Refactoring False Select a new Feature Yes In order to increase a defect detecting effectiveness of test cases created when using test driven development, we are proposing a modification to the standard TDD process flow, named TDD HQ - Higher Quality Testing in Test Driven Development, detailed in Figure 2. Based on the TDD process flow, several additional process steps were added. Choose Quality Improvement Aspect (A) Select Test Design Technique (B) Add a new Test Case No Yes All Q.I. Aspects Covered No More TCs for B in A A - Choosing Quality Improvement Aspect When testing software product, members of quality assurance teams investigate various aspects of product quality: functionality, performance, security, usability, robustness, etc. For them, it is important to cover both functional and nonfunctional quality features. However, developers tend to focus mainly on the functional aspects of software quality, as it was noticed in our previous empirical studies. This is why it is needed for a developer to explicitly choose an aspect of quality improvement during test driven development which will further guide designing of subsequent test cases. B - Selecting Test Design Technique Passed Execute all Test Cases Failed Make minimal code changes After deciding on the quality improvement aspect that should be in the focus of the current iteration, one of the appropriate test design technique should be selected. It is however important that this test design technique directly contribute to the previously chosen quality improvement aspect. In case there is a possibility to select two or more complementary test design techniques, developers could choose to iterate the process flow with the same quality improvement aspect but each time using a different test design technique for the same feature. Fig. 2. Stop Failed Execute all Test Cases Passed TDD HQ additions to standard TDD Process Flow Looking at the definition of TDD and its process flow, we can notice that there is no explicit guidance on how a developer should design and add a new test case. The reason for this problem is very obvious. TDD is a development practice and not a test design technique. Test data is usually formed from usage examples and serve as a safety net to detect potential miss-implementation of required functionality. In other words, test cases created using TDD could be considered as a by-product of the development process. But, often they are not. Many agile teams see the value in having a high number of automated test cases created at such an early (unit level) stage of software product development and consider those tests a considerable addition to the overall C - Check Whether More TCs for B in A Once the quality improvement aspect and the test design technique are determined, a classical red-green phase of TDD is conducted. Since a particular test design technique could require (by design) creation of several test cases, it is important to reflect if more test cases are needed for a given test design technique selected in B to satisfy a quality improvement aspect selected in A. If more test cases are needed, an additional red-green phase should be conducted for an each test case individually. D - Check Whether All Q.I. Aspects Covered After fully implementing a feature with the specific quality improvement aspect in mind and using the appropriate test design technique(s), developers could iterate through the implementation of the selected feature with a new A - B combination, or choose to continue with adding new features, if they exists. Sometimes, the decision on usage of the set of A-B combinations could be taken upfront, for all features by developer itself, or it could be enforced at the organisational level as a development policy implied to all development teams.

4 A. Combinations of A and B By using the general TDD HQ process flow (Figure 2), a specific instance of the process could be created purely based on the combinations of quality improvement aspects and test design techniques selected (A and B). Let us assume that a developer would like to use a classical TDD process for implementing a new feature. By using combinations provided in Table I, developers have a backward compatibility without any disturbance in the TDD process flow. Focusing on functionality as quality improvement aspect and using functional testing as a test design technique, developers can continue working using accustomed traditional TDD flow. If, on the other hand, developers would like to increase robustness of their software product, set of combinations provided in Table II could be one example of how to achieve such a goal. Those combinations are used as well in the empirical evaluation of our approach, detailed in section IV. TABLE II. TABLE I. A-B COMBINATION FOR CLASSICAL TDD Step A B 1 Functional Functional testing A-B COMBINATION FOR INCREASED ROBUSTNESS IN TDD Step A B 1 Functional Functional testing 2 Robustness Equivalence partitioning Number of possible A-B combinations and set of combinations a developer can follow in TDD HQ is quite high, considering that there is a large number of well known test design techniques ( [16]) and various quality improvement aspects to choose from. An additional research effort is needed to fully list, categorise and map sets of appropriate A-B combinations for each particular domain where TDD HQ could be fully utilised. B. Expectations and Threats Our main expectation of using TDD HQ instead of TDD, is in increased developers ability to create additional test cases designed more for the purpose of detecting defects. By doing this, developers would indirectly increase the quality of produced code and increase an overall quality of the final software product. However, there are possible threats to consider when augmenting TDD with different test design techniques: 1) Threats to Developers Productivity: By constantly switching between various quality improvement aspects and different test design techniques, developers may not be capable to have a firm focus on the actual functionality they are implementing. With gaining more experience, this threat may not have such a significance, but in the very beginning it might be advisable for a developer to pair-program with an experienced software tester. 2) Threats to Developers Efficiency: Some studies are suggesting that it takes more time to perform software development by using TDD instead of a traditional test-last approach. This was identified as one of the limiting factors of TDD adoption in our systematic literature review [8]. However, problem is that the time needed for corrective re-work (changing code based on failure reports) is usually not accounted and thus, on the long run, this should not be considered as a threat. A similar situation might occur with the usage of TDD HQ, as it requires some additional steps to be conducted in comparison to classical TDD. Again, on the long run, usage of TDD HQ might provide time-saving benefits. IV. EVALUATION OF THE PROPOSED APPROACH To better understand the potential improvements and the overall effect that our proposed approach might have on the software product quality, an empirical study was conducted in an academic environment. Although conducting experiment with students have a known set of validity threats and limitations on any conclusions drawn from it [17], we think it is a highly beneficial step in fully understanding the scope and limitations of any newly proposed process improvement technique. The following subsections elaborate on the design of the experiment, details of its execution and presents the analysis performed on the collected data. A. Experiment Design This experiment was setup with one goal in mind. We wanted to investigate if choosing set of A-B combinations for increased robustness in TDD (Table II) when following the TDD HQ process flow, will significantly increase the defect detecting effectiveness of the produced test cases. For that purpose, a following research question was defined: RQ1: Does the usage of TDD HQ improve defect detecting effectiveness of test cases? By a defect detecting effectiveness of test cases, we consider test case s ability to detect a defect in the code. This way, for each test case, we have a number of how many defects a particular test case can reveal. In most empirical studies performed by researchers, participants work on the solution for the same problem to minimise eventual validity threats and ease the process of analysis. This way academics have an opportunity to execute test cases of their participants against each others code, and thus gain much more inside information about the test cases effectiveness in finding defects. This is of great value especially when test driven development is the subject of study investigation, since the final set of test cases that accompany a software solution developed using TDD will show only its correctness but no defects in the same. In order to realistically measure the quality of testing we need to essentially have access to an ideal test suite which is capable of finding all the defects. Our approach here will be to approximate such an ideal test suite by combining all the test suites developed by several individual developers working on the same problem. Given such a set of multiple implementations and associated test suites, we are then able to cross-compare the ability of test cases to find defects. This is a similar approach we used in our previous empirical studies ( [12], [13], [18]). In order to test the goal of the experiment, with respect to the stated research question, the following null and alternative hypotheses were formulated:

5 Equivalence Partitioning Steps: 1) Identify the input space (all parameters that may affect the behavior). 2) Partition based on characteristics. Strategies: a) Individual partitioning of each input parameter b) Functional (output-based) modeling of the input space 3) Combine partitions into an input domain model. a) Here, identify and remove invalid combinations 4) Identify representative values 5) Derive test cases by complementing with expected output Fig. 3. H 1 0 H 1 a Equivalence Partitioning steps provided to experiment participants There is a difference in the quality of tests of TDD and TDD HQ participants. There is no difference in the quality of tests of TDD and TDD HQ participants. The focus of our experiment design is around the fact that it will be executed as part of a University course. Subjects of the experiment are master students admitted to the course on Software Verification & Validation (V&V) at Mälardalen University in Sweden during the autumn semester of The experiment was part of the hands-on laboratory assignment within the V&V course, and the subjects earned credits for the participation. Students were informed that the final grade for the course will be calculated only based on the written exam and their performance during the laboratory work will not affect the final grade. However, they had to fully complete their assignment. Similarly as in our previous studies, this experiment used a bowling game score calculator problem. The specification for this was based on the Bowling Game Kata. Participants in the TDD HQ group were instructed to use TDD in combination with equivalence partitioning test technique to develop software solutions. Figure 3 provide details on those instructions. Participants in the TDD group were instructed to use traditional TDD approach for software development. Instructions for TDD were given as prescribed by Flohr and Schneider [19]. Detailed information about the problem and instructions are provided on first authors webpage 1. To avoid problems with subjects unfamiliarity with the junit [20] testing framework and/or Eclipse IDE [21], a dedicated lecture on the usage of both in TDD was provided to students as well as several hands-on assignments. In addition, subjects were given an Eclipse project code skeleton with one simple test case at the beginning of the experiment. The reason for this is to enforce usage of the same programming interface to ease the process of the data analysis. Experiment participants were instructed, upon finalising their software implementation, to upload the source code together with the test cases (complete Eclipse workspace) to Blackboard, the course management tool. Additionally, participants had to complete a simple questionnaire stating their opinions on the quality of the provided solutions. 1 acc01/tddhq/ B. Execution Once all lectures within V&V course were completed, the pre-requirements for the experiment execution were in place. The total number of students participated in the last hands-on laboratory work in the course was 22. Students were informed that their solutions would be used for the experiment, but they were not provided any details on the goal of the experiment itself. Once again, we explicitly stated that their performance will not influence the final grade of the course. Subjects worked individually on the implementation and were randomly assigned to follow either TDD or TDD HQ approach. Their work was not time-boxed and subjects were given an opportunity to work on the implementation until they have enough quality confidence in the submitted solution. Since one way of measuring the quality of test cases was using a total number of failing assertions, a Java code skeleton was created and provided to subjects to enforce usage of same programming interface which would ease the process of executing test cases of subject X on the code of subject Y. After each student reported to researchers to have a complete solution created and saved in Blackboard, the test cases and the code for each student solution were extracted for the analysis. C. Analysis Table III presents total number of test cases created by our participants categorised by type of the test case and experiment group who created them. Looking at the overall number of test cases there is no significant difference, since TDD HQ group created around 5% less test cases than TDD group. However, it is interesting to observe that both groups created a slightly more number of positive than negative test cases leading to conclusion that positive test bias did not exist for both groups. One reason for such a result might be in the fact that during the course on Software Verification % Validation we emphasised much more on the negative aspect of software testing in comparison to the previous years course instances. TABLE III. TEST CASES DISTRIBUTION Test Type TDD Group TDD HQ Group Total Positive Negative Total Even though participants created almost equal number of positive and negative test cases, number of defects detected by those test cases is very different (shown in Table IV). By using just positive test cases from both groups we detected 355 defects, while negative test cases contributed with 1268 defects. This is one more evidence how valuable negative test cases are as part of any test suite, even if the positive test bias effect is not present. Looking at the differences between the groups we can observe that TDD HQ group detected around 17% more defects in total, but if we look only at the negative test cases, the increase in defect detection is around 20%. Our assumption is that this is a result of a systematic usage of equivalence partitioning test design technique within TDD HQ group of participants. With this study we are mainly focusing on the quality of produced test cases by measuring number of defects they

6 TABLE IV. DEFECTS DETECTED BY TEST CASES TABLE VI. DEFECT DETECTING EFFECTIVENESS OF TEST CASES Test Type TDD Group TDD HQ Group Total Positive Negative Total Test Type TDD Group TDD HQ Group Total Positive Negative Total can detect in the code. Since this code is also written by our participants, we can as well observe number of defects found in the code of our experiment participants groups. Table V presents the distribution of defects in the code of participants based on the group they belong to and type of tests which are detecting the defect. In total, code of TDD HQ group had around 26% less defects than code of TDD group. Only looking at the defects detected by negative tests, TDD HQ group had as much as 31% less than TDD group. Interestingly, more positive defects (around 8%) are detected in TDD HQ group. TABLE V. DEFECTS FOUND IN CODE Test Type TDD Group TDD HQ Group Total Positive Negative Total The number of defects detected with test cases provides some understanding of the quality of test cases, but considering the differences in the expertise levels of the developers, we would like to give a higher importance to a test case that is capable of finding defects in an implementation of a higher quality. This is why we need to calculate quality of code attribute for every developer (i) using next formula: Q code (i) = 1 N F T C(i) N T C where, N T C is a total number of test cases created by all developers and N F T C (i) represent total number of failing test cases on the code of a developer (i) when all test cases from all developers are executed. Once we calculated Q code value for each developer, we can now reward test cases who are able to detect defects in the underlying code. Quality of test cases for a developer (i) is calculated as a sum of the quality of each test case (j) from a set of test cases (n) of that developer (i): Q tests (i) = n Q T C (i.j) j=1 To calculate the quality of an individual test case (j) of a developer (i) we need to know on which developers code this test case is failing (m M). Sum of the (Q code ) values of those developers will define the quality of a particular test case (j): Q T C (i.j) = m Q code (k) k=1 In Table VI a summarised value for Q T C is presented categorised by type of test cases and the experiment groups. Once again we can see how negative test cases are contributing three times more than positive ones and 71% of the overall quality of testing score for both groups. When comparing between groups, TDD HQ group gathered 21% more quality score than TDD group. To have a better understanding of how different type of test cases contributes to the overall testing quality of each participant, Table VII shows quality score per test case values. With this table we can once again visualise how to a higher extent negative tests contributed to the overall testing quality. TABLE VII. Overall Positive Negative (a) TDD Group QUALITY SCORE PER TEST CASE VALUES Overall Positive Negative (b) TDD HQ Group This data is also used to address the research question RQ1. The Mann-Whitney nonparametric test was used in order to test the H 1 0 null hypotheses with α = We can not reject stated hypothesis (p-value is ), leading to the conclusion that there is no statistically significant difference in the quality of tests produced by participants of our experiment when following TDD and TDD HQ approach. The reason for this most likely lies in the fact that the sample size of participants used in our experiment is too small for the effect to be statistically significant. Unfortunately, sample size was not something we could influence on due to the fact that only students admitted to Software Verification & Validation course could participate in it. D. Validity Threats Similarly to several previously published experiments on TDD, this study was also performed in an academic setting. Therefore, external validity and the possibility to generalise the results of this study, is threatened with some known academic limitations: (i) using students as subjects, (ii) using small scale objects of investigation, and (iii) having short duration of the experiment. However, our main intention with this study was to gain more insights into the possible implications of TDD HQ as we were aware of these threats before conducting the study. By performing proper analysis of the data, we addressed potential threats of the internal validity of the study. As we could not assume normality of the data, we used nonparametric test to conduct hypotheses tests. However, it is without question a fact that, the sample size of the data was small, which is a major limitation for statistical analysis (and potentially also a cause for the inability for null hypothesis rejection). The only way to resolve this matter is through replications of the experiment. Construct validity refers to the correctness in the mapping between the theoretical constructs that are to be investigated,

7 and the actual observations of the study. Some of the constructs investigated in this study are not trivially defined, and may be subject to debate. For example, we defined quality of a test case as a form of calculating a defect detecting effectiveness, which is at the same time influenced by the quality of code where defect is actually detected. In order to mitigate this problem, we have provided detailed information on the calculation of each construct involved in the experiment. Reliability threats of this study (degree of certainty with which a replication could be performed) are addressed by providing sufficient information about the experiment, in the form of the experiment package and guidelines available for replication purposes. V. RELATED WORK Our work in this paper consists of two major aspects which should be considered when relating it to the current body of knowledge: (i) theoretical proposal for achieving higher quality of testing in test driven development - TDD HQ and (ii) empirical study designed to measure defect detecting effectiveness as a quality attribute of a test case. When looking at empirical studies with TDD as the main focus of investigation, we can observe that most of them are performed for the purpose of detecting potential benefits in the quality of the produced code. This was as well one of the finding in our systematic literature review [8] listing 48 empirical studies on TDD. One study did however had the focus on quality attributes of test cases when a test-first approach was used. Madeyski [22] investigated how usage of TDD can impact branch coverage and mutation score indicators. In his experiment, 22 students were divided in two groups: the test-first and the test-last, with the task of developing a web based conference paper submission system. This experiment shows no statistically significant differences in branch coverage and mutation score indicators, between the test-first and the test-last groups. In our previous study [12] we measured the code coverage and the mutation score indicators in a very similar way as it was done by Madeyski in his study. But, in accordance with the results of Madeyski, we were also not able to notice any differences between the experiment groups using the same quality indicators. This was one of the reason why we opted in the empirical study, presented in this paper, to use defect detecting effectiveness as a main attribute of differentiating participants quality of test cases. Defect detecting effectiveness was achieved by executing participants test cases on each others code, in a similar way as proposed by Goldwasser [23] for the purpose of better integration of software testing curriculum. We identified one additional study with the focus on developer s testing ability when following test driven development approach [24]. This was an industrial observational experiment where developers performed programming tasks in their own offices without the control of researchers. Once developers submitted their code and test cases, researchers performed mutation testing analysis to identify complementary test cases to the ones created during TDD process. Those unit tests, created by researchers, were still able to find several software faults in the developers submitted code. When comparing to the TDD HQ process flow, this approach could be simple utilised by developers if they would choose set of A-B combinations as presented in Table VIII. TABLE VIII. VI. A-B COMBINATION FOR MUTATION ANALYSIS IN TDD Step A B 1 Functional Functional testing 2 Mutation Metric Mutation Analysis CONCLUSIONS AND FUTURE WORK This paper present TDD HQ concept, an approach for achieving higher quality testing in test driven development. TDD HQ approach is focused on augmenting the standard TDD methodology with a suitable test design technique, for the purpose of satisfying a criteria of a specific quality improvement aspect we would like to achieve. Classical test driven development is designed in way to lead developers in writing of positive tests which will guide them forward in the implementation of a correct functionality. By using TDD HQ, developers do not necessarily focus only on verifying functionality, as it is defined in the given requirements, but they can as well increase security, robustness, performance and many other quality improvement aspects for the given software product, without interfering the classical red-greenrefactor flow. To exemplify this concept, we defined a set of quality improvement aspects and test design techniques combinations, for the purpose of improving robustness of the produced software solution. This was achieved by combining equivalence partitioning test design technique together with the test driven development. Initial evaluation of this approach was performed in an academic experiment, where the group who followed this approach created 5% less test cases but at the same time detected 17% more defects compared to a group of participants using classical test driven development. Such an increase in defects detection at an early stage of development (unit level) could have a very strong impact on the overall development process. Detecting defects at later stages (integration and system level testing) comes with a much greater cost [15]. However, since we could not show a statistically significant difference in the quality of test cases, between our group of participants, a new study needs to be conducted with a larger sample size. Since this empirical study investigated only one specific combination of quality improvement aspect (A) and accompanied test design technique (B), an additional research effort is needed to fully list, categorise and map sets of appropriate A-B combinations for each particular domain where TDD HQ could be fully utilised. Once this list is complete, a further investigation of its significance should be performed in the form of several empirical studies. Design of those studies should, in addition, account for investigation of the significance of possible threats when using TDD HQ, specifically threats to developers productivity and efficiency (as discussed in the section III-B). The proposed improvements (TDD HQ ) in the process flow of performing test driven development is expected to result in not just higher quality of produced tests but also in the higher quality and productivity in software systems, in terms of robustness and early defects detection.

8 ACKNOWLEDGMENT This work was supported by the Mälardalen Real-Time Research Center (MRTC) and SYNOPSIS project at Mälardalen University in Sweden. REFERENCES [1] K. Beck, Extreme Programming Explained: Embrace Change. Boston, MA, USA: Addison-Wesley Longman Publishing Co., Inc., [2] B. George and L. Williams, A Structured Experiment of Test-driven Development, Information and Software Technology, vol. 46, no. 5, pp , [3] H. Erdogmus, M. Morisio, and M. Torchiano, On the Effectiveness of the Test-First Approach to Programming, IEEE Transactions on Software Engineering, vol. 31, pp , [4] D. S. Janzen and H. Saiedian, On the Influence of Test-Driven Development on Software Design, Software Engineering Education and Training, Conference on, vol. 0, pp , [5] A. Gupta and P. Jalote, An Experimental Evaluation of the Effectiveness and Efficiency of the Test Driven Development, in Proceedings of the First International Symposium on Empirical Software Engineering and Measurement, ser. ESEM 07. Washington, DC, USA: IEEE Computer Society, 2007, pp [6] J. H. Vu, N. Frojd, C. Shenkel-Therolf, and D. S. Janzen, Evaluating Test-Driven Development in an Industry-Sponsored Capstone Project, in Proceedings of the 2009 Sixth International Conference on Information Technology: New Generations. Washington, DC, USA: IEEE Computer Society, 2009, pp [7] A. Causevic, D. Sundmark, and S. Punnekkat, An Industrial Survey on Contemporary Aspects of Software Testing, in Proceedings of the 3rd International Conference on Software Testing, Verification and Validation (ICST), 2010, pp [8] A. Causevic, D. Sundmark, and S. Punnekkat, Factors Limiting Industrial Adoption of Test Driven Development: A Systematic Review, in Software Testing, Verification and Validation (ICST), 2011 IEEE Fourth International Conference on, march 2011, pp [9] A. Causevic, D. Sundmark, and S. Punnekkat, Impact of Test Design Technique Knowledge on Test Driven Development: A Controlled Experiment, in XP, ser. Lecture Notes in Business Information Processing, C. Wohlin, Ed., vol Springer, 2012, pp [10] B. E. Teasley, L. M. Leventhal, C. R. Mynatt, and D. S. Rohlman, Why Software Testing Is Sometimes Ineffective: Two Applied Studies of Positive Test Strategy, Journal of Applied Psychology, vol. 79, no. 1, pp , [11] L. M. Leventhal, B. Teasley, D. S. Rohlman, and K. Instone, Positive Test Bias in Software Testing Among Professionals: A Review, in Selected papers from the Third International Conference on Human- Computer Interaction. London, UK: Springer-Verlag, 1993, pp [12] A. Causevic, S. Punnekkat, and D. Sundmark, Quality of Testing in Test Driven Development, in Quality of Information and Communications Technology (QUATIC), 2012 Eight International Conference on the, September [13] A. Causevic, R. Shukla, S. Punnekkat, and D. Sundmark, Effects of Negative Testing on TDD: An Industrial Experiment, in XP, ser. Lecture Notes in Business Information Processing, H. Baumeister and B. Weber, Eds., vol Springer, 2013 (to be presented). [14] Beck, Test Driven Development: By Example. Boston, MA, USA: Addison-Wesley Longman Publishing Co., Inc., [15] R. Megen and D. Meyerhoff, Costs and benefits of early defect detection: experiences from developing client server and host applications, Software Quality Journal, vol. 4, no. 4, pp , [16] L. Copeland, A Practitioner s Guide to Software Test Design. Norwood, MA, USA: Artech House, Inc., [17] D. I. K. Sjoberg, B. Anda, E. Arisholm, T. Dyba, M. Jorgensen, A. Karahasanovic, E. Koren, and M. Vokac, Conducting realistic experiments in software engineering, in Empirical Software Engineering, Proceedings International Symposium n, 2002, pp [18] A. Causevic, D. Sundmark, and S. Punnekkat, Test Case Quality in Test Driven Development: A Study Design and a Pilot Experiment, in Evaluation Assessment in Software Engineering (EASE 2012), 16th International Conference on, may 2012, pp [19] T. Flohr and T. Schneider, Lessons Learned from an XP Experiment with Students: Test-First Needs More Teachings, in Product-Focused Software Process Improvement, ser. Lecture Notes in Computer Science, J. Mnch and M. Vierimaa, Eds. Springer Berlin / Heidelberg, 2006, vol. 4034, pp [20] junit Framework, [21] Eclipse, [22] L. Madeyski, The impact of Test-First programming on branch coverage and mutation score indicator of unit tests: An experiment, Inf. Softw. Technol., vol. 52, pp , February [23] M. H. Goldwasser, A Gimmick to Integrate Software Testing Throughout the Curriculum, in Proceedings of the 33rd SIGCSE technical symposium on Computer science education, ser. SIGCSE 02. New York, NY, USA: ACM, 2002, pp [24] W. Shelton, N. Li, P. Ammann, and J. Offutt, Adding Criteria- Based Tests to Test Driven Development, in Proceedings of the 2012 IEEE Fifth International Conference on Software Testing, Verification and Validation, ser. ICST 12. Washington, DC, USA: IEEE Computer Society, 2012, pp [Online]. Available: