Goal-Oriented Measurement plus System Dynamics A Hybrid and Evolutionary Approach

Goal-Oriented Measurement plus System Dynamics A Hybrid and Evolutionary Approach Dietmar Pfahl Fraunhofer Institute IESE pfahl@iese.fhg.de Günther Ruhe University of Calgary ruhe@ucalgary.ca 1. Aim and Background This position paper aims at creating synergy from combining two powerful enabling techniques for software engineering decision support and software process improvement: the local, time-discrete measurement approach following the Goal/Question/Metric () paradigm [2], and the global, time-continuos simulation approach following the System Dynamics (SD) method [5, 6]. The purpose of the proposed hybrid and evolutionary approach is to enlarge the contributions of both methods when compared to its isolated application. In more detail, the purpose is to: Synchronize individual by putting them into a joint system perspective as suggested by SD; Enlarge by the dynamic features of SD; Incrementally increase the validity of both and SD models by mutually reusing and validating models generated from and SD; and Integrate real world empirical results with experiments in a virtual world offered by SD. Goal-oriented measurement following the paradigm is focusing on a top-down approach for definition of appropriate metrics and a bottom-up approach to analyze and interpret results. The approach leads to the specification and of a measurement program for a particular set of issues and will form the basis for the interpretation of the measurement data in the context of the precisely defined goal. Simulation in general has the ability to study complex systems in greater detail. The continuous simulation method SD is a very powerful approach with a broad range of applications in complex social, managerial, economic or engineering systems. As soon as an SD simulation model exists that reproduces current or past behavior of reality, systematic variation of model parameters (i.e., sensitivity analysis or inclusion and exclusion of model structures) can help in understanding, controlling, and improving system behavior. SD modeling is supported by comprehensive tool support. The underlying paradigm of SD is Systems Thinking [9]. The essential step toward Systems Thinking is to recognize the presence of feedback mechanisms in the observed system. In Systems Thinking, the behavior of a system is considered as primarily being generated by the interaction of all the feedback loops over time. 2. Limitations from the Isolated Application of and SD By applying, information is identified that is relevant to solve specific problems (goals), and that can be represented in a practical, applicable and interpretable way. By applying SD, decision-making support for global and dynamically complex planning and improvement problems is provided. However, there are some limitations of both and SD if used in isolation: (L1) Local versus Global is local in the sense that always one goal is investigated in isolation. For that purpose, other aspects and goals of the global software development projects are not taken into account. While it is important to understand certain aspects, it is hard to oversee its ramification for the whole system with its global objectives and constraints. It is not sufficient to improve certain parts of a system (to be locally optimal ) if this does not contribute and is not synchronized with all the remaining goals and constraints of software development and evolution ( globally optimal ). In contrast to models, SD models typically capture mutual influence and feedback of many interrelated parameters, though on a rather abstract ('global') level, i.e. without offering

means to analyze behavior of individual entities or attributes of entities. (L2) Static versus Dynamic is static and thus unable to reflect changes that are due to interactions and feedback causing change over time. What is missing to describe the dynamic changes in software development and evolution is a mechanism to integrate the models that reflect the status of a system at discrete points in time. On the other hand, calibration of time-dynamic SD models requires information from various types of static quantitative models (descriptive, predictive, and evaluative), and thus could benefit from the outcomes of measurement programs. (L3) Synchronization with other existing plans and models Typically, when modeling an aspect of software development or evolution, this is interacting with existing models or experience formulated for other purposes. In the same way, several individual plans are considered simultaneously. Unfortunately, there is no coordinating and integrating mechanism to model and describe the behavior of the overall system. This means, there is no mechanism to synchronize the plan with other existing models of software development and evolution. An appropriate SD model could serve as a mechanism facilitating such synchronization. (L4) Validity of models Development and validation of models is a timeconsuming and expensive process. Typically, from performing an individual measurement program, you learn about the current limitations of the model. Unfortunately, this process often takes too long and is too expensive. Again, a mechanism to check validity of models from a system-theoretic perspective would be very useful to improve efficiency of measurement-based improvement processes. (L5) Reuse of plans So far, reuse of plans is exclusively considered from a perspective of reusing certain parts of the template. However, there is a much larger potential of reuse if different models are considered simultaneously from a global/system-theoretic perspective. If timedynamic problems are addressed, with using suited SD models as a unifying shell, certain sub-models at the questions levels could be more easily reused, and a more comprehensive view on explanatory variables could be achieved. (L6) Validity of SD models Results from any simulation-based investigation, e.g., for the purpose of planning or exploration of improvement potentialities, are essentially based on the validity of the underlying models. However, there is no other way for checking the validity of SD models than conducting empirical investigations to compare real-word performance with virtual-world performance. would be a good means for effective planning and efficient conducting of appropriate empirical investigations. 3. How to Overcome Current Limitations: A Hybrid and Evolutionary Approach In what follows we describe the framework of a hybrid and evolutionary approach. It combines goal-oriented measurement with SD-based modeling and simulation. The main idea is to guide and supplement goal-oriented measurement by a system-oriented modeling and simulation approach. We consider software development from a system theoretic perspective and assume the existence of global (business) goals when studying systems. Starting from an initial understanding of the system under consideration, the most critical and worst understood parts of the overall system are most promising candidates for an in-depth investigation. This investigation can be done by designing and implementing a -based measurement program with the goal derived from the subsystem under investigation. The results from this program are used to better understand system structure and system behavior at this point. This process can be iterated several times, where not only one measurement program needs to be considered at each step. The application of this interactive (between SD and ) and evolutionary (models, results, and insights are evolving and relying on each other) process results in a sequence of SD models with increasing accuracy and validity in describing reality, a sequence of plans derived from the global improvement goal(s) and the perspective of the whole system, a means to incrementally improve the validity of both the SD models and the models by checking their respective consistency, a sequence of reusing parts of both the SD models (information flow) and the models

(influencing variables on explanatory variables) for the respective other model, a method to combine the results of with the power of SD to show how different individual/local goals fit together, a method that combines experiments in a virtual world by conducting simulation runs with experiments in a real world by performing goaloriented measurement. 4. Complementary Elements of SD-based Modeling and -based Measurement SD-based modeling and simulation helps analyze real world problems that involve dynamic (time-dependent) behavior, while -based measurement aims to quantitatively describe real world phenomena or develop (static) quantitative models for prediction or evaluation [4]. -based measurement programs, i.e., the collected data and resulting (static) models, can be utilized at various stages of the SD modeling process. Four (informal) relationship patterns between SD-based modeling and -based measurement can be identified (cf. [7] for a comprehensive discussion of relationships between process models, static quantitative models, and SD models). The development and application of SD models involves the following modeling and simulation products: SD goal definition: formally defines the problem to be analyzed; specifies scope, dynamic focus, purpose, user, and environment of the related SD model. SD reference mode: describes current problematic or intended (improved) future dynamic behavior of interest; during SD modeling, the reference behavior serves for validating initial versions of the SD model. SD base mechanism: describes a causal relationship between attributes of real-world entities; base mechanisms are graphically represented by directed arcs that connect one dependent with one or more independent variables; the impact of a dependent variable on the independent variable can be either positive or negative. SD causal diagram: integrates the minimal set of SD base mechanisms that is able to reproduce the SD reference mode; this typically results in a directed graph with cycles (feedback loops). SD flow graph: transforms the structural information contained in the SD causal diagram into a semi-formal graphical SD representation language; specifies the mathematical structure of the SD model (number, type and form of the mathematical equations). SD model equation: defines an element of the SD flow graph (level, rate, constant, and auxiliary) in terms of a mathematical equation. SD simulation result: comprises the outputs from simulation runs and their interpretation with respect to the SD goal definition. The definition and performance of -based measurement programs involves the following planning and measurement products: goal definition: formally defines the measurement goal; specifies object, quality focus, purpose, viewpoint, and environment of the related measurement program. plan: brakes down the goal definition into questions, and defines measures (metrics) and (static) mathematical models needed to answer these questions. abstraction sheet: specifies types and scales of one dependent variable and associated independent variables (variation factors) for each (static) mathematical model; in addition, hypotheses are formulated with regards to the measurement results of the dependent variable and the impact of the variation factors. measurement plan: defines the roles, templates and tools involved in data collection activities; specifies the assignment of persons to roles, the logical synchronization of data collection activities with the software development process, and the schedule of data collection activities. measurement result: comprises the measurement data and their interpretation with respect to the goal definition. Figure 1 shows relationship pattern time-series data. Descriptive models that plot quantitative data along the time-axis (time-series data) are a very useful means to precisely describe the dynamic behavior of those realworld entities/attributes that are in the focus of interest. The relevant set of these data plots defines the SD reference mode.

SD Goal Definition SD Goal Definition SD Reference Mode Goal Definition SD Reference Mode Goal Definition SD Base Mechanism Plan SD Base Mechanism Plan Real World SD Causal Diagram Abstraction Sheet Real World SD Causal Diagram Abstraction Sheet SD Flow Graph Measurement Plan SD Flow Graph Measurement Plan SD Model Equation Measurement Result SD Model Equation Measurement Result SD Simulation Result SD Simulation Result Figure 1: SD- relationship pattern time-series data Figure 2 shows relationship pattern simple predictive or evaluation model. SD base mechanisms are those fundamental cause-effect relationships between realworld entities/attributes that are supposed to generate the reference behavior captured by the SD reference mode. Simple predictive or evaluation models describing the functional relationship between one dependent variable and one or more independent variables typically represent such cause-effect relationships. Because SD base mechanisms describe only qualitative cause-effect relationships, i.e., the exact form of the functional relationship is not yet needed, data analysis methods such as classification trees, rough sets, statistical correlation analysis or statistical testing can be sufficient to derive the required models from empirical data. SD Goal Definition Figure 3: SD- relationship pattern hierarchically structured predictive or evaluation model Figure 4 shows relationship pattern parameter estimation and calibration. The specification of SD model equations typically involves the reuse or development of quantitative models, i.e. descriptive, predictive or evaluation models. In order to derive the precise functional form, statistical parameter estimation techniques and methods such as regression analysis can be applied. Due to the systematic integration of models into SD models, simulation modeling according to SD can be considered the dynamic expansion of the well-established approach ("Dynamic "). The following section illustrates with the help of an example how modeling and SD modeling can be smoothly integrated by exploiting the SD- relationship patterns 1-4. SD Reference Mode Goal Definition SD Goal Definition SD Base Mechanism Plan SD Reference Mode Goal Definition Real World SD Causal Diagram Abstraction Sheet SD Base Mechanism Plan SD Flow Graph SD Model Equation SD Simulation Result Measurement Plan Measurement Result Figure 2: SD- relationship pattern simple predictive or evaluation model Real World SD Causal Diagram SD Flow Graph SD Model Equation SD Simulation Result Abstraction Sheet Measurement Plan Measurement Result Figure 3 shows relationship pattern hierarchically structured predictive or evaluation model. plans can be considered a means for hierarchically structuring Figure 4: SD- relationship pattern parameter estimation and calibration predictive or evaluation models (nesting). 5. Integration of and System Dynamics An Example Assume the task to develop a simulation model for the purpose of understanding and controlling software project dynamics generated by trade-off effects between effort consumption, project duration, and product quality ("magic triangle").

Following a top-down approach, a sequence of SD models is developed that integrate at each development stage one or more models. First, a sequence of three SD models (abbreviated by SD[1a], SD[1b], SD[1c]) is developed that illustrate the integration of models according to the various SD- relationship patterns. The first set of models focuses on the effort-duration trade-off only. In order to illustrate how locally defined models can be integrated into a SD model that covers all trade-off relations comprised in the magic triangle, the development of a second set of SD models (SD[2a], SD[2b]) is presented (cf. [1] for a more comprehensive SD model). The purpose of SD models SD[1a] to SD[1c] is to understand and control the dynamics of software projects with regards to effort consumption and duration. Exploiting SD- relationship pattern "time-series data", the SD reference mode, representing the typical project progress over time, is defined by the first model abbreviated by [1] (cf. Figure 5). [1]: software product completion pattern over time (time-series data / S-shaped curve) [3]: effort size dependency (simple predictive model / COCOMO [3]) Effort = a * Size b [4]: duration effort size dependency (hierarchically structured predictive model / COCOMO) Duration = c * Effort d Eventually, models [2] to [4] are used to define causal relationships and SD model equations. The stepwise integration of the models into the initial SD model SD[1a] (cf. Figure 7) is shown in Figures 9 (SD model SD[1b]) and 11 (SD model SD[1c]). Characterization of SD model SD[1a] (cf. Figure 7): State variable: SW_Product Input parameters: number of requirements, duration, requirements to LOC transformation factor # requirements KLOC requ to loc transformation facto Project Completion 1 % development rate SW Product total duration 1 % Project Duration Figure 5: Product completion pattern (S-shaped) In order to define the SD base mechanisms and the SD causal diagram, the following models [2], [3], and [4] are used. [2]: manpower allocation pattern over time (descriptive model) Persons Graph Lookup - design manpower allocation pattern 2 Figure 7: Initial SD model The initial SD model shown in Figure 7 does not yet contain any model resulting from empirical analysis. It simply assumes that the development rate is calculated as follows: (*) development_rate = KLOC / total_duration The dynamic change of state variable SW_Product is determined by (*) and the input variables. The simulation result of model SD[1a] is shown in Figure 8. 2 Graph for SW Product 1 Time 2 Figure 6: Typical manpower allocation pattern 2 4 6 8 1 12 14 16 18 2 SW Product : Current-mod1 Figure 8: Simulation output - development progress

# requirements design pages requ to design transformation factor design manpower allocation pattern design rate KLOC average design productivity per person SW Design requ to loc transformation factor actual design manpower allocation design effort average design manpower <KLOC> total effort & rework rate design & rework duration manpower allocation pattern design schedule completion average & rework productivity per person SW total duration design effort factor actual & rework manpower design duration allocation factor average & rework manpower start time <KLOC> <total effort> & rework effort & rework schedule completion <total duration> nominal effort rate & rework duration test & rework allocation pattern average effort factor productivity per person SW Product duration factor actual manpower allocation average test & rework manpower effort test start time schedule completion nominal test & rework effort <total effort> duration <total duration> effort factor nominal test & rework duration duration factor As the behavior pattern of SW Product does not show similarity to the S-shaped pattern of the reference behavior (cf. Figure 5), model validity can be doubted. By including model [2], a more realistic behavior can be achieved (cf. Figure 1). Characterization of model SD[1b] (cf. Figure 9): State variable: SW Product Input parameters: number of requirements, duration, requirements to LOC transformation factor. # requirements development manpower Model 2 allocation pattern KLOC development rate requ to loc transformation factor schedule completion SW Product total duration Figure 9: Initial SD model integrating [2] 2 1 Graph for SW Product 2 4 6 8 1 12 14 16 18 2 SW Product : Current-mod2 # requirements Model 3 development manpower allocation pattern KLOC development rate requ to loc transformation factor schedule completion SW Product total duration coeff a coeff b total effort coeff c coeff d Model 4 Figure 11: Initial SD model integrating [3], [4] In order to prepare for the inclusion of quality related aspects, and in particular the inclusion of quality assurance activities throughout all development phases, model SD[2a] refines model SD[1c] such that individual project phases (e.g., design,, test) are explicitly represented. Characterization of model SD[2a] (cf. Figure 12): State variables: SW Design, SW, SW Product Input parameters: number of requirements, requirements to design pages factor, requirements to LOC transformation factor Integrated models (selection): [5]: phase specific manpower allocation patterns (descriptive model) [6]: typical time overlap between phases (descriptive model) [7]: typical effort split between phases (simple predictive model) Figure 1: Simulation output - development progress By integrating models [3] and [4], new causal relationships between product size, effort consumption and project duration are inserted into the SD model (cf. Figure 11). This eliminates the need for having "duration" as model input. coeff a coeff b coeff c coeff d Characterization of model SD[1c] (cf. Figure 11): State variable: SW Product Input parameters: number of requirements, requirements to LOC transformation factor Design Implementation Test Figure 12: Refined SD model with separate design, and test phases

<SW > <undetected and uncorrected defects in undetected defects code> in product undetected defects code defect density Injected code defect in code in code injection rate Product defect Detected injection undetected factor defects code defects per code defect Injected code defect in code KLOC in detection rate in test injection rate <SW > < defect & rework rate> undetected and injection factor code defects per test defect uncorrected defects in KLOC in detection factor <SW > code defect detection rate in Detected code < <actual & & rework rate> rework manpower (code allocation> undetected inspection) <# code and < uncorrected defects in inspections> rate> code defect detection rate in Detected code <actual & rework manpower (code Corrected Product test defect detection and allocation> inspection) code defect correction Corrected <# code correction effectiveness rate in product defect inspections> defect correction rate in test detection factor (code Inspections Corrected code inspection code defect correction inspection effectiveness) rate in rate defect < & < defect correction factor rework rate> test defect effort> <total effort> < detection factor (code correction factor KLOC per code code inspection Inspections < optimal KLOC per effort factor> inspection effectiveness) & rework inspection code inspection rate duration> defect < & correction factor rework rate> < KLOC per code <average optimal KLOC per & rework inspection & rework manpower> code inspection <SW > duration> <average & rework manpower> <SW > undetected and uncorrected defects in product code defect density in product <SW > Figures 13 and 14 show simulation results generated by model SD[2a]. 6 18 18 Design Test Implementation Input parameters: number of requirements, requirements to design pages factor, requirements to LOC transformation factor, number of design inspections, number of code inspections Integrated models (selection): [8]: defect detection during design (simple predictive model) 2 4 6 8 1 12 14 16 18 2 des_insp_eff = f (pages_per_insp, des_defect_density) [9]: defect detection during (simple predictive model) SW Design : Current SW : Current SW Product : Current code_insp_eff = f (LOC_per_insp, code_defect_density) Figure 13: Simulation output - design, code, and product development progress [1]: defect detection during test (simple predictive model) The overlap between phases results from model [6]. The various phase-specific manpower allocation patterns result from the combination of models [5] to [7]. Design Test Implementation test_eff = f (test_effort, code_defect_density) 8 Defect Co-Flow Defect Detection in Design Defect Detection in Implementation Defect Detection in Test 4 2 4 6 8 1 12 14 16 18 2 actual manpower allocation : Current actual design manpower allocation : Current "actual & rework manpower allocation" : Current "actual manpower allocation" : Current Figure 14: Simulation output - manpower allocation Model SD[2b] adds the defect generation-detectioncorrection co-flow to the software design-test workflow of SD[2a], and thus integrates product quality aspects with effort and duration aspects of the project dynamics. Characterization of SD model SD[2b] (cf. Figure 15): State variables: SW Design, SW, SW Product, Design Inspections, Design Injected, Design Detected, Design Corrected, Inspections, Injected, Detected, Corrected, Product Detected, Product Corrected. Development Flow Design Implementation Test Figure 15: Refined SD model with defect co-flow Figures 16 and 17 show simulation results generated by model SD[2b]. 6 3 3 2 1 4 1 2 3 4 4: # of code inspections = 14 3: # of code inspections = 1 2: # of code inspections = 3 1: # of code inspections = 2 4 6 8 1 12 14 16 18 2 Figure 16: Simulation output - actual manpower allocation for and test phases Figure 16 shows how the performance of code inspections during affects the actual and test manpower allocations. It can be

seen, that there is a shift of effort from the test phase to the phase if code inspections are conducted. 8 4 Graph for undetected defects in product 2 4 6 8 1 12 14 16 18 2 undetected defects in product : Current undetected defects in product : Current1 undetected defects in product : Current2 undetected defects in product : Current3 1 2 3 4 4: # of code inspections = 14 3: # of code inspections = 1 2: # of code inspections = 3 1: # of code inspections = Figure 17: Simulation output - undetected defects in the product Figure 17 shows the effect of conducting code inspections on product quality (number of undetected defects). Observing the results of cases 1-3 one can see that product quality increases with the number of code inspections. This is partly due to the fact that code inspection effectiveness depends on the average code size subject to inspection (in case 3: ca. 18 LOC per inspection), as implied by [9]. On the other hand, it can be observed that the product quality in case 4 (14 code inspections) is worse than in case 3 (1 code inspections). This is due to the fact that the reduction of test effort has a negative effect on test effectiveness, as implied by [1]. When the number of inspections is above a certain value, the negative effect exceeds the positive effect of conducting code inspections. 6. Discussion In software development, we are typically encountered with situations of uncertainty, incompleteness, fuzziness or even inconsistency. We can not expect reliable data of models for making any kind of decision related to products, processes,, tools, or resources. We need an evolutionary, learning-based approach combined with an appropriate and sound methodology to enable intelligent decision support [8]. In this paper, we purpose a new hybrid method of called Dynamic. It main purpose is to organize mutual interaction and support between sequences of models from System Dynamics and Goal-oriented measurement. The subsequent models are related to each other. Their increasing quality results from the supplementary character of the two types of models: System Dynamics addresses the behavior and performance of systems from a global perspective. In the same way, is local in the sense that always one goal is investigated in isolation. From the integration of both types of models, new insights can be gained from experiments in both the real world and the virtual world (conducting simulation). The benefits of Dynamic are three-fold: It integrates and synchronizes individual models from a global system perspective; It incrementally increases the validity of both and SD models by mutually reusing and validating models generated from SD and, respectively; It is a means to integrate real-world empirical results with experiments in a virtual world offered by SD. The proposed approach was illustrated by the example of evolving models for trade-off analysis between effort, time, and quality. The principal example has shown that the isolated application of either one of the two approaches would not have been able to provide the same quality of models and meaningfulness of insights. Future research will be devoted to further formalize Dynamic. This will lead to operational artifacts, rules and guidelines how to proceed. Further effort will be devoted to provide tool support and to adapt and validate the approach in an industrial environment. REFERENCES [1] Abdel-Hamid TK, Madnick SE, Software Projects Dynamics an Integrated Approach, Prentice-Hall, 1991. [2] Basili VR, Caldiera G, Rombach HD, Van Solingen R, Goal Question Metric Paradigm, in: Marciniak JJ (ed.), Encyclopedia of Software Engineering, Vol. 1, pp. 578-583, John Wiley & Sons, 21. [3] Boehm BW, Software Engineering Economics, Prentice Hall, 1981. [4] Briand LC, Differding CM, Rombach HD, Practical Guidelines for Measurement-Based Process Improvement, Software Process Improvement and Practice 2 (4), pp. 253-28, 1996. [5] Coyle RG, System Dynamics Modelling A Practical Approach, Chapman & Hall, 1996. [6] Forrester JW, Industrial Dynamics, Productivity Press, 1961.

[7] Pfahl D, An Integrated Approach to Simulation- Based Learning in Support of Strategic and Project Management in Software Organisations, Stuttgart: Fraunhofer IRB Verlag, PhD Theses in Experimental Software Engineering, Vol. 8, 21. [8] Ruhe G, "Software Engineering Decision Support - A New Paradigm for Learning Software Organizations", appears in Proc. 4th Workshop on Learning Software Organizations, Chicago, Springer 23. [9] Senge P, The Fifth Discipline the Art & Practice of the Learning Organization, New York: Doubleday, 199.