Trace-Context Sensitive Performance Profiling for Enterprise Software Applications

Transcription

1 Trace-Context Sensitive Performance Profiling for Enterprise Software Applications Presentation at SPEC International Performance Evaluation Workshop 2008 Matthias Rohr 1, André van Hoorn 1, Simon Giesecke 2, Jasminka Matevska 1, Wilhelm Hasselbring 1, and Sergej Alekseev 3 1 Graduate School TrustSoft, Software Engineering Group, University of Oldenburg, Germany 2 OFFIS Institute for Information Technology, Oldenburg, Germany 3 Nokia Siemens Networks GmbH & Co KG, Berlin, Germany Contact: rohr(ät)informatik.uni-oldenburg.de, July 2, 2008 Matthias Rohr et al. (TrustSoft, Univ. Oldenburg) Trace-Context Sensitive Performance Profiling June 27, 2008, Darmstadt Germany 1 / 29

2

3

4

5

6

7

8

9

10

11

12

13

14 Outline Foundations 1 Motivation 2 Foundations 3 Approach to Calling-context Sensitive Timing Behavior Modeling 4 Related work 5 Evaluation 6 Conclusions and Future Work Matthias Rohr et al. (TrustSoft, Univ. Oldenburg) Trace-Context Sensitive Performance Profiling June 27, 2008, Darmstadt Germany 5 / 29

15 Monitoring Foundations <<Component>> M :Catalog :Bookshop :CRM :Catalog M M M M <<Component>> :Bookshop M <<Component>> :CRM M M M M M Monitoring of Response times: Time between start and end of software operation executions Execution sequences of operations for each thread Monitoring tool and trace analysis tool: Kieker 1 [Rohr et al., 2008b] Uses Aspect-oriented Programming (AspectJ) Matthias Rohr et al. (TrustSoft, Univ. Oldenburg) Trace-Context Sensitive Performance Profiling June 27, 2008, Darmstadt Germany 6 / 29

16 Monitoring of traces Foundations Operation TraceID t in t out rt = t out t in a c b c c Matthias Rohr et al. (TrustSoft, Univ. Oldenburg) Trace-Context Sensitive Performance Profiling June 27, 2008, Darmstadt Germany 7 / 29

17 Monitoring of traces Foundations Operation TraceID t in t out rt = t out t in a c b c c Trace reconstruction Traces are sequences of executions that correspond to system service requests Sequence diagrams and dynamic call trees represent (classes) of traces Matthias Rohr et al. (TrustSoft, Univ. Oldenburg) Trace-Context Sensitive Performance Profiling June 27, 2008, Darmstadt Germany 7 / 29

18 Foundations Construction of dynamic call trees Dynamic call trees: Dynamic call trees [Ammons et al., 1997] represent execution sequences by an ordered tree. a A c C D E F d e A.a c d f C.c D.d E.e F.f C.c D.d E.e e Matthias Rohr et al. (TrustSoft, Univ. Oldenburg) Trace-Context Sensitive Performance Profiling June 27, 2008, Darmstadt Germany 8 / 29

19 Foundations Construction of dynamic call trees Dynamic call trees: Dynamic call trees [Ammons et al., 1997] represent execution sequences by an ordered tree. a A c C D E F d e A.a c d f C.c D.d E.e F.f C.c D.d E.e e Matthias Rohr et al. (TrustSoft, Univ. Oldenburg) Trace-Context Sensitive Performance Profiling June 27, 2008, Darmstadt Germany 8 / 29

20 Foundations Construction of dynamic call trees Dynamic call trees: Dynamic call trees [Ammons et al., 1997] represent execution sequences by an ordered tree. a A c C D E F d e A.a c d f C.c E.e F.f C.c D.d E.e e Matthias Rohr et al. (TrustSoft, Univ. Oldenburg) Trace-Context Sensitive Performance Profiling June 27, 2008, Darmstadt Germany 8 / 29

21 Foundations Construction of dynamic call trees Dynamic call trees: Dynamic call trees [Ammons et al., 1997] represent execution sequences by an ordered tree. a A c C D E F d e A.a c d f C.c D.d E.e F.f C.c D.d E.e e Matthias Rohr et al. (TrustSoft, Univ. Oldenburg) Trace-Context Sensitive Performance Profiling June 27, 2008, Darmstadt Germany 8 / 29

22 Foundations Construction of dynamic call trees Dynamic call trees: Dynamic call trees [Ammons et al., 1997] represent execution sequences by an ordered tree. a A c C D E F d e A.a c d f C.c D.d E.e F.f C.c D.d E.e e Matthias Rohr et al. (TrustSoft, Univ. Oldenburg) Trace-Context Sensitive Performance Profiling June 27, 2008, Darmstadt Germany 8 / 29

23 Foundations Construction of dynamic call trees Dynamic call trees: Dynamic call trees [Ammons et al., 1997] represent execution sequences by an ordered tree. a A c C D E F d e A.a c d f C.c D.d E.e F.f C.c D.d E.e e Matthias Rohr et al. (TrustSoft, Univ. Oldenburg) Trace-Context Sensitive Performance Profiling June 27, 2008, Darmstadt Germany 8 / 29

24 Foundations Construction of dynamic call trees Dynamic call trees: Dynamic call trees [Ammons et al., 1997] represent execution sequences by an ordered tree. a A c C D E F d e A.a c d f C.c D.d E.e F.f C.c D.d E.e e Matthias Rohr et al. (TrustSoft, Univ. Oldenburg) Trace-Context Sensitive Performance Profiling June 27, 2008, Darmstadt Germany 8 / 29

25 Foundations Construction of dynamic call trees Dynamic call trees: Dynamic call trees [Ammons et al., 1997] represent execution sequences by an ordered tree. a A c C D E F d e A.a c d f C.c D.d E.e F.f C.c D.d E.e e Matthias Rohr et al. (TrustSoft, Univ. Oldenburg) Trace-Context Sensitive Performance Profiling June 27, 2008, Darmstadt Germany 8 / 29

26 Foundations Construction of dynamic call trees Dynamic call trees: Dynamic call trees [Ammons et al., 1997] represent execution sequences by an ordered tree. a A c C D E F d e A.a c d f C.c D.d E.e F.f C.c D.d E.e e Matthias Rohr et al. (TrustSoft, Univ. Oldenburg) Trace-Context Sensitive Performance Profiling June 27, 2008, Darmstadt Germany 8 / 29

27 Outline Approach to Calling-context Sensitive Timing Behavior Modeling 1 Motivation 2 Foundations 3 Approach to Calling-context Sensitive Timing Behavior Modeling Calling-context Equivalence Classes Optimization of the Trace-context Sensitive Timing Behavior Model 4 Related work 5 Evaluation 6 Conclusions and Future Work Matthias Rohr et al. (TrustSoft, Univ. Oldenburg) Trace-Context Sensitive Performance Profiling June 27, 2008, Darmstadt Germany 9 / 29

28 Approach to Calling-context Sensitive Timing Behavior Modeling Approach Overview Goal Derivation of a timing behavior model with lower variance and less multi-modality in timing behavior distributions Matthias Rohr et al. (TrustSoft, Univ. Oldenburg) Trace-Context Sensitive Performance Profiling June 27, 2008, Darmstadt Germany 10 / 29

29 Approach to Calling-context Sensitive Timing Behavior Modeling Approach Overview Goal Derivation of a timing behavior model with lower variance and less multi-modality in timing behavior distributions Definition: Calling-context of an operation call Set of circumstances or facts that surround an operation call, in particular the sequence of surrounding operation executions. Matthias Rohr et al. (TrustSoft, Univ. Oldenburg) Trace-Context Sensitive Performance Profiling June 27, 2008, Darmstadt Germany 10 / 29

30 Approach to Calling-context Sensitive Timing Behavior Modeling Approach Overview Goal Derivation of a timing behavior model with lower variance and less multi-modality in timing behavior distributions Definition: Calling-context of an operation call Set of circumstances or facts that surround an operation call, in particular the sequence of surrounding operation executions. Steps of the approach 1 Integrate calling-context information into timing behavior model 2 Optimize timing behavior model; e.g. model size reduction Matthias Rohr et al. (TrustSoft, Univ. Oldenburg) Trace-Context Sensitive Performance Profiling June 27, 2008, Darmstadt Germany 10 / 29

31 Approach to Calling-context Sensitive Timing Behavior Modeling Calling-context Equivalence Classes Calling-context equivalence Three levels of abstraction for calling-context information: Equivalence relations on software operation executions Two executions of the same operation are caller-context equivalent (cp. Graham et al. [1982]) := called from operations with the same name. stack-context equivalent (cp. Ammons et al. [1997]) := equal paths from their tree nodes to root trace-context equivalent := 1) corresponding trees are equal 2) tree nodes have same position in tree Matthias Rohr et al. (TrustSoft, Univ. Oldenburg) Trace-Context Sensitive Performance Profiling June 27, 2008, Darmstadt Germany 11 / 29

32 Approach to Calling-context Sensitive Timing Behavior Modeling Calling-context Equivalence Classes Calling-context equivalence Three levels of abstraction for calling-context information: Equivalence relations on software operation executions Two executions of the same operation are caller-context equivalent (cp. Graham et al. [1982]) := called from operations with the same name. stack-context equivalent (cp. Ammons et al. [1997]) := equal paths from their tree nodes to root trace-context equivalent := 1) corresponding trees are equal 2) tree nodes have same position in tree Stack-context equivalence caller-context equivalence Trace-context equivalence stack-context equivalence Matthias Rohr et al. (TrustSoft, Univ. Oldenburg) Trace-Context Sensitive Performance Profiling June 27, 2008, Darmstadt Germany 11 / 29

33 Approach to Calling-context Sensitive Timing Behavior Modeling Calling-context Equivalence Classes Caller-context equivalence (cp. Graham et al. [1982]) Two executions of the same operation are caller-context equivalent if they are called from operations with the same name. A.a C.c D.d E.e F.f C.c D.d E.e Caller context equivalence for E.e B.b A.a F.f C.c D.d E.e B.b B.b C.c D.d E.e Dynamic call tree for trace 1 Dynamic call tree for trace 2 Figure: Example: Caller-context equivalence for operation E.e (3 eq. classes) Matthias Rohr et al. (TrustSoft, Univ. Oldenburg) Trace-Context Sensitive Performance Profiling June 27, 2008, Darmstadt Germany 12 / 29

34 Approach to Calling-context Sensitive Timing Behavior Modeling Calling-context Equivalence Classes Stack-context equivalence (cp. Ammons et al. [1997]) Two executions of the same operation are stack-context equivalent if the paths from the corresponding nodes to its root are equal. A.a C.c D.d E.e F.f C.c D.d E.e Stack context equivalence for E.e B.b A.a F.f C.c D.d E.e B.b B.b C.c D.d E.e Dynamic call tree for trace 1 Dynamic call tree for trace 2 Figure: Example: Stack-context equivalence for operation E.e (4 eq. classes) Matthias Rohr et al. (TrustSoft, Univ. Oldenburg) Trace-Context Sensitive Performance Profiling June 27, 2008, Darmstadt Germany 13 / 29

35 Approach to Calling-context Sensitive Timing Behavior Modeling Calling-context Equivalence Classes Trace-context equivalence Example Two executions of the same operation are trace-context equivalent if the corresponding trees are equal and the both executions correspond to dynamic call tree nodes with the same position within the tree. A.a C.c D.d E.e F.f C.c D.d E.e Trace context equivalence for A.a B.b A.a F.f C.c D.d E.e B.b B.b C.c D.d E.e Dynamic call tree for trace 1 Dynamic call tree for trace 2 Figure: Example: Trace-context equivalence for operation A.a (2 eq. classes) For both traces together, number of trace-contexts = total number of nodes in non-equal trees = 19. Matthias Rohr et al. (TrustSoft, Univ. Oldenburg) Trace-Context Sensitive Performance Profiling June 27, 2008, Darmstadt Germany 14 / 29

36 Approach to Calling-context Sensitive Timing Behavior Modeling Calling-context Equivalence Classes Calling-Context Sensitive Timing Behavior Model Monitored response times of all instrumented software operations RT = [... ] Partitioning based on operation name equality Partitioning based on caller context equivalence Partitioning based on stack context equivalence Partitioning based on trace context equivalence Calling-context sensitive timing behavior model derived from monitoring data: Complete partitioning of all response times based on trace-context, stack-context, or caller-context equivalence Matthias Rohr et al. (TrustSoft, Univ. Oldenburg) Trace-Context Sensitive Performance Profiling June 27, 2008, Darmstadt Germany 15 / 29

37 Approach to Calling-context Sensitive Timing Behavior Modeling Optimization of the Trace-context Sensitive Timing Behavior Model Additional Model Optimization using Tree Operators Possibly new problems resulting from calling-context analysis: Efficiency: Too many calling-contexts Calling-contexts do not differ in timing behavior distributions Applicability and robustness of statistical methods Calling-contexts with an insufficient number of measurements Matthias Rohr et al. (TrustSoft, Univ. Oldenburg) Trace-Context Sensitive Performance Profiling June 27, 2008, Darmstadt Germany 16 / 29

38 Approach to Calling-context Sensitive Timing Behavior Modeling Optimization of the Trace-context Sensitive Timing Behavior Model Additional Model Optimization using Tree Operators Possibly new problems resulting from calling-context analysis: Efficiency: Too many calling-contexts Calling-contexts do not differ in timing behavior distributions Applicability and robustness of statistical methods Calling-contexts with an insufficient number of measurements Leaf nodes without siblings are removed Leaf nodes without a sufficient amount of observations are linked to an ancestor node Leaf nodes with similar distribution characteristics are merged Matthias Rohr et al. (TrustSoft, Univ. Oldenburg) Trace-Context Sensitive Performance Profiling June 27, 2008, Darmstadt Germany 16 / 29

39 Outline Related work 1 Motivation 2 Foundations 3 Approach to Calling-context Sensitive Timing Behavior Modeling 4 Related work 5 Evaluation 6 Conclusions and Future Work Matthias Rohr et al. (TrustSoft, Univ. Oldenburg) Trace-Context Sensitive Performance Profiling June 27, 2008, Darmstadt Germany 17 / 29

40 Related work: Related work Correlating control-flow information and performance measurements: Graham et al. [1982] introduced the profiler gprof that can evaluate measurements in the context of the caller Ammons et al. [1997] introduced evaluation in the context of the stack of callers; Implemented by commerical profilling tools (e.g., Intel s VTune Profiler) Handling clusters in response time distributions: Bulej et al. [2005] describe clusters in timing behavior measurements and apply k-means clustering on response time data Matthias Rohr et al. (TrustSoft, Univ. Oldenburg) Trace-Context Sensitive Performance Profiling June 27, 2008, Darmstadt Germany 18 / 29

41 Outline Evaluation 1 Motivation 2 Foundations 3 Approach to Calling-context Sensitive Timing Behavior Modeling 4 Related work 5 Evaluation Field Study Case Study 6 Conclusions and Future Work Matthias Rohr et al. (TrustSoft, Univ. Oldenburg) Trace-Context Sensitive Performance Profiling June 27, 2008, Darmstadt Germany 19 / 29

42 Field Study Evaluation Field Study Setting System at Nokia Siemens Networks Commercial software platform for implementing telecommunication network services (e.g., signaling) Workload: Test driver of Nokia Siemens Networks (5, ,000 Calls/Hour) Goals Demonstration of applicability in a complex system Demonstration of resolving multi-modality and standard deviation reduction Matthias Rohr et al. (TrustSoft, Univ. Oldenburg) Trace-Context Sensitive Performance Profiling June 27, 2008, Darmstadt Germany 20 / 29

43 Results Evaluation Field Study Response time in microseconds :18 14:20 14:22 14:24 Calendar time (hour:minute) (a) Response times measured for operation f. Probability density Response time in microseconds (b) Probability density distribution for operation f. Figure: Clusters and multi-modality for operation f. Matthias Rohr et al. (TrustSoft, Univ. Oldenburg) Trace-Context Sensitive Performance Profiling June 27, 2008, Darmstadt Germany 21 / 29

44 Evaluation Field Study Results Removal of multi-modality Probability density Response time in microseconds Probability density Probability density Trace context 1 Trace context Response time in microseconds (a) PDF for trace-context Response time in microseconds (b) PDF for trace-context 1 and 3. Matthias Rohr et al. (TrustSoft, Univ. Oldenburg) Trace-Context Sensitive Performance Profiling June 27, 2008, Darmstadt Germany 22 / 29

45 Evaluation Field Study Results Removal of multi-modality Probability density Probability density Trace context 1 Trace context Response time in microseconds (c) PDF for trace-context Response time in microseconds (d) PDF for trace-context 1 and 3. Total average decrease in standard deviation ; 2.20; % reduction of standard deviation Matthias Rohr et al. (TrustSoft, Univ. Oldenburg) Trace-Context Sensitive Performance Profiling June 27, 2008, Darmstadt Germany 22 / 29

46 Outline Evaluation Case Study 1 Motivation 2 Foundations 3 Approach to Calling-context Sensitive Timing Behavior Modeling 4 Related work 5 Evaluation Field Study Case Study 6 Conclusions and Future Work Matthias Rohr et al. (TrustSoft, Univ. Oldenburg) Trace-Context Sensitive Performance Profiling June 27, 2008, Darmstadt Germany 23 / 29

47 Evaluation Case Study Case Study Goals and research questions Setting Quantitative evaluation of standard deviation reduction. Is trace-context analysis more effective than caller- and stack-context analysis? How does the number of monitoring points relate to the number of calling-contexts? System: ibatis JPetStore 5 Web application deployed on Apache Tomcat Workload: Probabilistic workload generated by Markov4JMeter a [van Hoorn et al., 2008] 10 concurrent users Experiment run statistics: Random instrumentation: of 199 possible monitoring points Experiment runs last 18 minutes 2 2,032,572 monitored operation executions per run a Matthias Rohr et al. (TrustSoft, Univ. Oldenburg) Trace-Context Sensitive Performance Profiling June 27, 2008, Darmstadt Germany 24 / 29

48 Evaluation Case Study Case study: Standard deviation reduction for two particular instrumentations Average st.dev. decrease in % 18 mon.pts. scenario Full instrumentation Caller-context analysis Stack-context analysis Trace-context analysis Decrease of stdev in % Decrease of stdev in % Caller Stack Trace Calling context type (e) 18 mon.pts. scenario CallerContext StackContext TraceContext Calling context type (f) Full instrumentation Matthias Rohr et al. (TrustSoft, Univ. Oldenburg) Trace-Context Sensitive Performance Profiling June 27, 2008, Darmstadt Germany 25 / 29

49 Evaluation Case Study Case study: Standard deviation reduction for two particular instrumentations Average st.dev. decrease in % 18 mon.pts. scenario Full instrumentation Caller-context analysis Stack-context analysis Trace-context analysis Decrease of stdev in % Decrease of stdev in % Caller Stack Trace Calling context type (g) 18 mon.pts. scenario CallerContext StackContext TraceContext Calling context type (h) Full instrumentation A significant part of the variance is connected to trace-context information! Matthias Rohr et al. (TrustSoft, Univ. Oldenburg) Trace-Context Sensitive Performance Profiling June 27, 2008, Darmstadt Germany 25 / 29

50 Evaluation Case Study Case study: Number of monitoring points vs. standard deviation reduction Average relative st.dev. decrease in % Trace context analysis Stack context analysis Caller context analysis 1st and 3rd quartile Number of monitoring points Figure: Average decrease in standard deviation for different numbers of monitoring points using calling-context information compared to standard deviation using no calling-context information. Matthias Rohr et al. (TrustSoft, Univ. Oldenburg) Trace-Context Sensitive Performance Profiling June 27, 2008, Darmstadt Germany 26 / 29

51 Evaluation Case Study Case study: Number of monitoring points vs. number of calling-contexts Number of contexts Stack contexts Caller contexts Number of contexts Trace contexts Stack and Caller contexts Number of monitoring points Number of monitoring points (a) (b) The number of stack-contexts and number of caller-contexts both grow linearly with a similar rate (cp. [Rohr et al., 2008a]). The number of trace-contexts increases faster than the number of stack-contexts and number of caller-contexts. Matthias Rohr et al. (TrustSoft, Univ. Oldenburg) Trace-Context Sensitive Performance Profiling June 27, 2008, Darmstadt Germany 27 / 29

52 Summary Conclusions and Future Work Contribution: New approach to consider trace-context information in timing behavior models Trace-context equivalence extends the concepts of caller-context equivalence and stack-context equivalence Results of the field study: Trace-context analysis is applicable in large complex software system Demonstration of multi-modality removal and standard deviation reduction Results of the case study: Significant reduction of standard deviation in timing behavior model Trace-context analysis is more effective than stack- and caller-context analysis The number of trace-contexts grows linearly Matthias Rohr et al. (TrustSoft, Univ. Oldenburg) Trace-Context Sensitive Performance Profiling June 27, 2008, Darmstadt Germany 28 / 29

53 Conclusions and Future Work Outlook and Future work Current limitations / problems: Explosion of the model size is still possible Multi-modality and high variance may not be an issue in a performance analysis Outlook and possibilities for future work: Efficiency evaluation: Benefit vs. monitoring costs Evaluation for execution times (instead response times) Demonstration of effectiveness in the context of anomaly detection ([Rohr et al., 2007]) Considering also asynchronous communication within traces Acknowledgement We would like to acknowledge Nokia Siemens Networks Berlin, Business Service Solution for supporting this project. Matthias Rohr et al. (TrustSoft, Univ. Oldenburg) Trace-Context Sensitive Performance Profiling June 27, 2008, Darmstadt Germany 29 / 29

54 References G. Ammons, T. Ball, and J. R. Larus. Exploiting hardware performance counters with flow and context sensitive profiling. In Proceedings of the Conference on Programming language design and implementation (PLDI 97), pages ACM, ISBN doi: / L. Bulej, T. Kalibera, and P. Tůma. Repeated results analysis for middleware regression benchmarking. Performance Evaluation, 60(1-4): , ISSN doi: /j.peva S. L. Graham, P. B. Kessler, and M. K. McKusick. gprof: a call graph execution profiler. SIGPLAN Notes, 17(6): , doi: / M. Rohr, S. Giesecke, and W. Hasselbring. Timing Behavior Anomaly Detection in Enterprise Information Systems. In J. Cardoso, J. Cordeiro, and J. Filipe, editors, Proceedings of the Ninth International Conference on Enterprise Information Systems (ICEIS 07), volume DISI, pages INSTICC Press, June ISBN M. Rohr, A. van Hoorn, S. Giesecke, J. Matevska, and W. Hasselbring. Trace-context sensitive performance models from monitoring data of software systems. In C. Lebsack, editor, Proceedings of the Workshop on Tools Infrastructures and Methodologies for the Evaluation of Research Systems (TIMERS 08) at IEEE International Symposium on Performance Analysis of Systems and Software 2008, pages 37 44, Apr. 2008a. Matthias Rohr et al. (TrustSoft, Univ. Oldenburg) Trace-Context Sensitive Performance Profiling June 27, 2008, Darmstadt Germany 29 / 29

55 Additional slides M. Rohr, A. van Hoorn, J. Matevska, N. Sommer, L. Stoever, S. Giesecke, and W. Hasselbring. Kieker: Continuous monitoring and on demand visualization of Java software behavior. In Proceedings of the IASTED International Conference on Software Engineering 2008, pages ACTA Press, Feb. 2008b. ISBN A. Sabetta and H. Koziolek. Measuring performance metrics: Techniques and tools. In I. Eusgeld, F. Freiling, and R. Reussner, editors, Dependability Metrics, volume 4909 of Lecture Notes in Computer Science (LNCS). Springer, ISBN A. van Hoorn. Workload-sensitive timing behavior anomaly detection in large software systems, Sept Master s thesis (Diplomarbeit), Department of Computing Science, University of Oldenburg, Germany. A. van Hoorn, M. Rohr, and W. Hasselbring. Generating probabilistic and intensity-varying workload for web-based software systems. In S. Kounev, I. Gorton, and K. Sachs, editors, Performance Evaluation Metrics, Models and Benchmarks: Proceedings of the SPEC International Performance Evaluation Workshop (SIPEW 08), volume 5119 of Lecture Notes in Computer Science (LNCS), pages , Heidelberg, June SPEC, Springer. ISBN doi: / Matthias Rohr et al. (TrustSoft, Univ. Oldenburg) Trace-Context Sensitive Performance Profiling June 27, 2008, Darmstadt Germany 29 / 29

56 Additional slides Application scenario: Anomaly Detection Response time in milliseconds WISAD: Anomaly WISAD: no anomaly Period of anomaly injection Excluded excution Experiment time in seconds Matthias Rohr et al. (TrustSoft, Univ. Oldenburg) Trace-Context Sensitive Performance Profiling June 27, 2008, Darmstadt Germany 29 / 29

57 Additional slides Standard deviation reduction Probability density Response time in milliseconds Probability density Response time in milliseconds Probability density Response time in milliseconds Quantitative evaluation metric: Standard deviation of trace-contexts weighted by invokation frequency compared to original standard deviation: 1 P n i=1 TC i stdev(tc i ) Original stdev(original) Standard variation relates to uncertainty and confidence in statistical analysis For operation insertorder() example: Standard deviation original distribution: 100% Standard deviation trace-context sensitive model: 61.27% = 38.73% less Matthias Rohr et al. (TrustSoft, Univ. Oldenburg) Trace-Context Sensitive Performance Profiling June 27, 2008, Darmstadt Germany 29 / 29

58 Additional slides Resolving multi-modality 1/3 Operation neworder() has a multi-modal response time distribution: Probability density Response time in milliseconds Figure: Kernel density estimate for all response times of operation neworder() Matthias Rohr et al. (TrustSoft, Univ. Oldenburg) Trace-Context Sensitive Performance Profiling June 27, 2008, Darmstadt Germany 29 / 29

59 Additional slides Resolving multi-modality 2/3 Trace-context analysis replaces the original distribution by two uni-modal distributions: Probability density Response time in milliseconds Probability density Probability density Response time in milliseconds Response time in milliseconds Matthias Rohr et al. (TrustSoft, Univ. Oldenburg) Trace-Context Sensitive Performance Profiling June 27, 2008, Darmstadt Germany 29 / 29

60 Additional slides Resolving multi-modality 3/3 Identified by trace analysis: Two trace-contexts One caller-context and one stack-context The two trace-contexts correspond to two call scenarios: $ ActionServlet OrderBean $ ActionServlet OrderBean OrderService process(..) process(..) neworder() neworder() insertorder(..) (a) (b) The sequence diagrams only distinguish in the call action to insertorder(). Matthias Rohr et al. (TrustSoft, Univ. Oldenburg) Trace-Context Sensitive Performance Profiling June 27, 2008, Darmstadt Germany 29 / 29

61 Calling-Context Tree Additional slides Calling-Context Tree Monitored response times of all instrumented software operations RT = [... ] Partitioning based on operation name equality Partitioning based on caller context equivalence Partitioning based on stack context equivalence Partitioning based on trace context equivalence Complete calling-context sensitive timing behavior model A complete timing behavior model consists of any node subset of the CCT that corresponds to a complete partitioning of all monitored observations. Matthias Rohr et al. (TrustSoft, Univ. Oldenburg) Trace-Context Sensitive Performance Profiling June 27, 2008, Darmstadt Germany 29 / 29

62 Additional slides Influences to Software Timing Behavior System architecture: Hardware design Software design System usage [cp. Sabetta and Koziolek, 2008]: Workload intensity Concurrent service requests Number of active users [van Hoorn, 2007] Service demand characteristics Parameter values Parameter value size Caller identity / stack content / history of calls System state: Cache content Load balancer state Other active processes on same platform Database content Matthias Rohr et al. (TrustSoft, Univ. Oldenburg) Trace-Context Sensitive Performance Profiling June 27, 2008, Darmstadt Germany 29 / 29

63 Additional slides Nokia Siemens Field Study Figure: UML Sequence Diagrams from a module of a partially instrumented telecommunication signaling system of Nokia Siemens Networks derived from monitoring data. (Operation names changed, operations omitted). Matthias Rohr et al. (TrustSoft, Univ. Oldenburg) Trace-Context Sensitive Performance Profiling June 27, 2008, Darmstadt Germany 29 / 29

64 Additional slides Nokia Siemens Field Study Figure: Tree representation of each of the traces. Probability density Trace context 1 Trace context 3 Matthias Rohr et 200 al. (TrustSoft, 300 Univ. Oldenburg) 400 Trace-Context 500 Sensitive Performance Profiling 65 June 27, 2008, Darmstadt 70 Germany / 29 Probability density