Slinger Jansen Utrecht University Utrecht University - Slinger Jansen. 3 Utrecht University - Slinger Jansen

Transcription

1 So what about case studies then? Case Study Research On the research validity of organizational case studies An intensive, detailed description and analysis of a single project or program in the context of its environment. An uncontrolled (prospective or retrospective) observational study involving an intervention and outcome in a single patient. (Also known as a single case report or anecdote.) Type of research which focuses on one subject, or person, as opposed to a group of subjects. A written account of an engineering activity as it was actually carried out; it s a true story, narrative, with detailed backgrounds such as: calculations, drawings, budget, and schedule elements. Slinger Jansen Utrecht University 2 Case Study Properties Case Study Types Focus on one case, thus hard to generalize conclusions There are three different case study types according to Yin Generally highly detailed Must be seen in its own context at all times Anecdotal Descriptive: The case study aims to describe the object of study, without any aim to prove or disprove, but with clear distinction of what the object of study is ipad Mini: What does a person use an ipad Mini for? Exploratory: The case study aims to explore the object of study, to define the borders around the object of study, and is usually the beginning of new research ipad Mini: How does the technology within an iphone relate to the technology within a ipad Mini? Explanatory: The case study aims to prove or disprove some causal relationship within the object of study ipad Mini: Why does a person buy the ipad Mini over a Galaxy Tab? A case study can be of multiple types 3 4 Three Basic Steps in Case Study Design (Yin, 2004) Case Study Designs 1. Define the case, e.g. CCU practices at Exact Software The use of a pattern at Stabiplan Requirements management at Planon 4 types of designs (based on a 2x2 matrix) Single-case vs. Multiple-case design 2. Single or multiple? Holistic vs. Embedded design Is this one black sheep we re studying? Do we want to generalize (at least a bit)? Does it fit the research resources? Is it secretly already a multiple-case? Multiple patterns at one company? 3. Use theory development or not? Go into it blankly, or with hypotheses (from an accept notification: The main shortcoming with regard to research method is that they lack explicit formulation of a strong hypothesis and that they do not follow a clear hypothesis-driven approach to research and research documentation (cf. Yin, 2002, on case study research). 4. Perform quality case screening Is this really the best case for you? Will you be able to generalize some of your findings based on this unique set of cases? Basic Types of Designs for Case Studies (Yin, page 40) 5 1

2 Establish a Chain of Evidence Validity Case study research questions Protocol -> How do you answer your questions? Case study database 4 primary types of validity Internal Validity External Validity Reliability Case study report 7 Seeks an agreement between a theoretical concept and a specific measuring device, such as observation. Establishing correct operational measures for the concepts being studied If your research question is Why are Samsung phones so sexy? Sexy a feeling slightly below the abdomen (lower belly) of humans that is related to the liking of an object. Can only be measured by applying a human being as a measuring instrument. Common signals are Widening op pupils Nausia Higher pulse Are we measuring what we intend to measure? Akin to the requirements problem: are we building the right system? If we don t get this right, the rest doesn t matter Constructs: abstract concepts Theoretical constructions Must be operationalized in the experiment Necessary condition for successful experiment Divide construct validity into three parts: In English: Saliva production (aka drooling) Define your concepts in such a way that you can derive conclusions from them Make sure that no room for error is left (uncovered) in the measurements Intentional Validity Representation Validity Observation Validity 9 Intentional Validity Representation Validity Do the constructs we chose adequately represent what we intend to study Are our constructs specific enough? Do they focus in the right direction? Eg, is it intelligence or cunningness? How well do the constructs or abstractions translate into observable measures Two primary questions: Do the sub-constructs properly define the constructs? Do the observations properly interpret, measure or test the constructs? 2 ways to argue for representation validity Face validity (that is, looks appropriate ) Very weak argument! Strengthened by consensus of experts Content validity Check the operationalization against the domain for the construct The extent to which the tests measure the content of the domain being tested - ie, cover the domain Both are qualitative judgments 2

3 Observation Validity How good are the measures themselves? Different aspects illuminated by Predictive validity Criterion validity Concurrent validity Convergent validity Discriminant validity Predictive Validity Observed measure predicts what it should predict and nothing else E.g., college aptitude tests are assessed for their ability to predict success in college Criterion Validity Degree to which the results of a measure agree with those of an independent standard Eg, for college aptitude, GPA or successful first year Concurrent Validity The observed measure correlates highly with an established set of measures Eg, shorter forms of tests against longer forms Convergent Validity Observed measure correlates highly with other observable measures for the same construct Utility is not that it duplicates a measure but is a new way of distinguishing a particular trait while correlating with similar measures Discriminant Validity The observable measure distinguishes between two groups that differ on the trait in question Lack of divergence argues for poor discriminant validity Internal Validity Internal Validity (aka Conclusion Validity) Can we be sure our results really follow from the data? Have we adequately ruled out rival hypotheses? Have we eliminated confounding variables? Participant variables Researcher variables Stimulus, procedural and situational variables Instrumentation Nuisance variables Confounding sources of internal invalidity H: History events happen during the study (eg, 9/11) M: Maturation older/wiser/better between during study I: Instrumentation change due to observation/measurement instruments S: Selection differing nature of participants effects of choosing participants (1) The rigor with which the study was conducted (e.g., the study's design, the care taken to conduct measurements, and decisions concerning what was and wasn't measured) Establish that a certain condition is caused and only caused by the conditions you defined. (2) the extent to which the designers of a study have taken into account alternative explanations for any causal relationships they explore (Huitt, 1998). In studies that do not explore causal relationships, only the first of these definitions should be considered when assessing internal validity. 16 Internal Validity External Validity Demonstrating that certain conditions are in fact the cause of other conditions That is, conditions not mentioned or studied are not the actual cause Example: if a study concludes that X causes Y without knowing some third factor Z may have actually caused Y, the research design has failed to deal with threats to internal validity Internal validity applies to explanatory and causal studies only, not to descriptive or exploratory studies It is important to challenge any inferences you make during your study as any incorrect inferences may detract from internal validity Two positions The generalizability of the causal relationship beyond that studied/observed Are the findings generalizable beyond the immediate case study? Eg, do studies of very large reliable real-time systems generalize to small.com companies? The extent to which the results support the claims of generalizability Eg, do the studies of 5ESS support the claim that they are representative of realtime ultra reliable systems Case studies have been criticized for offering a poor basis for generalization This is contrasting case studies (which rely on analytical generalization) with survey research (which relies on statistical generalization), which is an invalid comparison Generalization of the theory must be tested by replicating the findings over several different cases. 3

4 Reliability Reliability/ Repeatability Demonstrating that the operations of a study can be repeated with the same results Note: the repetition of the study should occur on the same case, not replicating the results on a different case The goal of reliability is to minimize the errors and biases in a study A prerequisite for reliability testing is documented procedures for the case study A good guideline is to perform research so that an auditor could follow the documented procedures and arrive at the same results Demonstrate that the operations of a study such as the data collection procedures can be repeated with the same results The same results will be found if someone performs the same research again with the same subject. The research is still valid when applying the same research approach to another case. 20 Tactics to Address Quality in Case Study Design External Validity (Generalisation) Establish a domain to which the study s findings can be generalized In English: We expect to find similar results for similar case studies with another study subject Case Study Tactics for the Four Design Tests (Yin, page 34) 22 Other kinds of validity Content validity - The extent to which a measurement reflects the specific intended domain of content (Carmines & Zeller, 1991, p.20). I.e., did you take the right data set into account and did you not miss anything important? Convergent Validity - The general agreement among ratings, gathered independently of one another, where measures should be theoretically related. Dependability - Being able to account for changes in the design of the study and the changing conditions surrounding what was studied. Discriminate Validity - The lack of a relationship among measures which theoretically should not be related. Face Validity - How a measure or procedure appears

5 Gathering Evidence Document Study Operational documents Archives , etc. Interviews Much of the same impact factors as for surveys open ended questions closed questions interview protocol, etc. Direct observations During case study you ll encounter many interesting phenomena Participant observation Taking part in the process Observation influences the end result Physical artifacts Physical goods Software artifacts etc. Takehome: case study research benefits from multiple sources of evidence, but this does require a more advanced method of data analysis Evidence, but what next? Triangulation is an essential part of a case study Pattern Matching the (mathematically) expected and predicted evidence is similar to the actual evidence Data source triangulation, when the researcher looks for the data to remain the same in different contexts; Explanation Building iterative and narative way of building up an explanation Investigator triangulation, when several investigators examine the same phenomenon; Time-Series Analysis when you measure certain data over time, it is possible to derive conclusions from it Theory triangulation, when investigators with different view points interpret the same results; and Logic Models similar to explanation building, yet now supported by formal logic Methodological triangulation, when one approach is followed by another, to increase confidence in the interpretation. Cross-Case Synthesis earlier research shows X in case study A in similar manner such that X can also be found in case study B Note: not specific to case studies Common Case Study Questions Myths about Case Study Research How do I know if I should use the case study method to do my study? How should I select the case to be studied? I am studying a software company. What is my case Is it the developers? The bug tracking system? The whole company? How much time and effort should I devote to collecting the case study data? How do I know whether I m finished collecting the data? How do I start analyzing my case study data? General, theoretical (context-independent) knowledge is more valuable than concrete, practical (context-dependent) knowledge. Myths! One cannot generalize on the basis of an individual case; therefore, the case study cannot contribute to scientific development. The case study is most useful for generating hypotheses; that is, in the first stage of a total research process, whereas other methods are more suitable for hypothesis testing and theory building. The case study contains a bias toward verification, that is, a tendency to confirm the researcher s preconceived notions. It is often difficult to summarize and develop general propositions and theories on the basis of specific case studies. See: Flyvbjerg, B.; Five Misunderstandings about Case Study Research. Qualitative Inquiry 12 (2) , April 2006]

6 The Case Study in Practice Product Software Formulate research questions (for the complete DELIVER project) What are the problems DELIVER must deal with? (Raison d etre) Software products are different from conventional products Easily malleable after delivery by: Producer How have these problems been formulated in literature? (Q1) What problems are currently not solved for software vendors? (Q2) What are possible contributions of DELIVER to Third party developer Specific solution for a generic environment Science (Q3) Industry (Q4) Determine research methods: Literature study (Q1 - Q4) Experimental (Q3, Q4) Case studies (Q2, Q4) Deliver... Software depends on different components, which are equally changeable Deliver project studies software release, delivery, and deployment By explicitly managing component knowledge Through case studies (approx 15% of deployments fail) Case studies show complex scenarios: AutoCAD Plug-in developer Multiple deployments on load balancing servers (with 100% uptime) Natural observation (Q4) -> MS, Adobe, etc Configuration Updating Consists of the following processes: release delivery deployment usage and activation Different software vendors apply different techniques and methods to support these processes Nobody has an overview, except us! Our studies are actually multi-case studies. Allows: Comparison Generalization of our results Supplies sources for industrial cooperation 33 Deployment Delivery Release Activation & Usage Organization SW Development Software Configuration Management System Sales Vendor Release Repository Product R1 Product R2. Product Rn Deployment Support Licenses Usage Support Relationship Management System Advertise Update Deliver Update Deployment Feedback License(s) Usage Feedback Uninformed Receive Info Informed Receive Update Possesses Update Deploy/Install Update Configure Installed Activate Update Reconfigure Activated Remove Rollback/ Deinstall Deactivate Case Studies at Product Software Vendors Case Studies at Product Software Vendors Thus, a number of research questions have been formulated A case study approach is created (20 pages) that First defines the concepts Find out practices and techniques used by product software vendors Establish problem fields as to encourage research in those areas Then defines the evidence gathering methods Defines how these methods relate to the concepts Describes how conclusions can be derived from possible evidence Provides a detailed procedure on the full case study (planning, etc) Candidates are selected from a sample of the industry (yes, we know this is not perfect) Propose process improvements based on literature and other cases Find lacking tools Research Questions: A case study database is created for each candidate (even those who are not interested) Candidates are presented with the case study approach RQ1: How does CCU of a product software vendor relate to the Deliver models? On a green light, the case study starts according to it s scenario Subquestion 1: What does CCU look like for the vendor? Evidence is gathered and stored in the case study database (interviews, etc, approximately 100 hours of research at vendors) Evidence is evaluated and a report is written describing all relevant evidence Conclusions are drawn according to the protocol in the same report Subquestion 2: How does CCU of this vendor compare to the Deliver models? Subquestion 3: How does CCU of the vendor relate to previous case studies? Subquestion 4: What improvements can be made to CCU of the vendors? Results are published in famous magazines, journals, and of course at conferences at exotic locations! The scientific community is a little richer once more!

7 Sources of Information The Cases Interviews Two sessions of 45 minutes per person People are selected through management and relevant job descriptions Software study Attempt to install and update a number of versions Document study Select and read documentation describing the process, double check during second session of interviews Add to case database Direct observations Browsing through the bug reporting system Spending some time at contracts department etc. No participation observations Case Identification Number of No of formal informal Organization Duration of Code Time interviewees interviewees size study ERPComp Early months OCSComp Early ,5 months HISCComp Mid weeks FMSComp Late weeks CMSComp Early weeks TDSComp Mid weeks 37 MULTI-CASE STUDY DATABASE MULTI-CASE STUDY DATABASE CASE STUDY DATABASE CASE STUDY DATABASE CASE STUDY PROTOCOL Research execution Case Study Invitation Software and document study STUDY REPORTS Send formal to network Send to personal netw. INVITATION Interviews INTERVIEW REPORTS Provide presentation to respondents Initialization INTRODUCTORY PRESENTATION CASE STUDY APPROACH DOCUMENT On-site observation Document study Software study OBSERVATION REPORTS RELEVANT DOCUMENTS SOFTWARE USE REPORTS [else] [approval] Communicate protocol Discuss, create, and sign NDA Determine Project Champion NON-DISCLOSURE AGREEMENT Material selection Peer review SELECTED MATERIAL CASE STUDY REPORT Determine project scope Determine first sources and access levels SCOPE DEFINITION INITIAL SOURCES [else] [approval] Case Study Champion Review [else] [approval] MULTI-CASE STUDY DATABASE Validity (1/2) Case Study Finalization Case study report finalization CASE STUDY REPORT CASE STUDY DATABASE Construct validity: are the concepts being studies operationalized and measured correctly? Created a glossary of terms Case study result presentation at host organization FINAL PRESENTATION Used the well-established CCU model Periodic peer review of the case study database Create technical report Add case study database to multi-case database TECHNICAL REPORT NUMBER the site Inform the participants of the aim of the research (i.e., well prepared elevator pitch) Aftercare Internal validity: identify the causal relationships Review case study reports by informed outsiders Review by project champion Technique of explanation building in the report 7

8 Validity (2/2) Case study protocol External validity: establish whether the results of the research can be generalized Case selection from Platform for Productsoftware Different companies of different sizes Active in different areas and markets List of members of PPSW shows these are dead in the middle Emperical reliability: the research can be repeated by others with similar result Define a CASE STUDY PROTOCOL Define a case study database Use a universal case study report format Serves three purposes: Define aims to avoid confusion and conflict (used as a tool for negotiation) Convince the participants to participate Used to instruct different researchers at different sites Kept it short Listed potential participants to demonstrate impact on host organization Used as a completeness checklist at end of research Validity criteria a guiding beacon for this research 1. Introduction describing the case study and motivivation in short 2. Research describing the research questions, method and vision 2.1 Research project 2.2 Conceptual model and terminology 2.3 CCU model description 2.4 Research questions 2.5 Research methods 4. Description of CCU at the host organization 3. Description of the host organization 3.1 Short description 3.2 Main product(s) 3.3 Employees and organizational structure 3.4 s 3.5 The market 4.1 Development management and organization Versioning Documentation Software architecture The host s software knowledge mgmt processes 4.2 Release process 4.3 Delivery process 4.4 Deployment process 4.5 Usage and activation 5. Observations and Conclusions 6. Potential Improvements Digital organization materials Presentations Documents Manuals Licenses Etc. Digital research materials Research protocol Case study report Demo screenshots and movies s Etc. Paper organization materials Presentations Documents Manuals Licenses Etc. Paper research materials Signed NDAs Interview notes Observation notes Etc. The best case The worst case 2003: First exploratory case Organization with many customers, but only a limited number with CCU problems Immediately hit jackpot, highly mature organization with willingness to participate Organization not interested in CCU 160,000 customers Project champion left halfway through the project Project champion had special position within the company Participants wary of project Novel phenomena worty of reporting Results marginally relevant Better screening would have solved all this 8

9 Taken from a colleague: Case Selection Taken from a colleague: Case Approach (1) We focused on cases in four different companies. All of them were successful small- or medium-sized software product companies. These companies were successful in that they showed high profit margins of 20% or more from 2004 to 2008, grew from 45% to 600% during the four-year period, and had no debt. Three of them had an AAA credit rating. The companies were selected based on accessibility through our research partnership. In companies B and C there were two divisions, one where deployment was not considered problematic and another in which deliveries were causing some frustration. In both cases, the division that considered deliveries more problematic was selected for the study. In both cases, the other division primarily operated in a COTS fashion, whereas the studied divisions had a more complex product or environment that impacted their deliveries. This research was conducted as a case study [15] of four Finnish software product companies. It is considered an embedded case study with four units of analysis, i.e., the four case organizations. Initially, an interview guide consisting of focus areas and open questions was prepared. The guide was based on prior work concerning the software deployment process [2] and our experience at quality goal workshops with the companies [9]. Then we interviewed two persons from each organization who had responsibilities in software deployment. During the interview, one researcher was mainly responsible for asking questions (first author) while the other researcher made notes (second author). Occasionally, the researcher making notes would ask questions if he felt that a topic was not properly covered. Mäntylä M. V. and Vanhanen J., "Software Deployment Activities and Challenges An Industrial Case Study of Four Software Product Companies", in Proceedings of the 16th European Conference on Software Maintenance and Reengineering, (CSMR), Taken from a colleague: Case Approach (2) Taken from a Colleague: Validity (1) After the interview, the following six (!) steps were taken. 1. First, immediately after the two interviews in a case organization, the first author wrote a memo about that case based on his fresh recollections and notes. This memo was reviewed by the second author. 2. Second, after the interviews had been transcribed by a third-party service, the researchers did a thematic coding of the interviews using ATLAS.ti software. This coding formed an initial set of activities, goals, challenges and context variables. 3. Third, complete case summaries were made and presented to the interviewees. 4. Fourth, a spreadsheet was created to summarize all the activities and context variables of each case. 5. Fifth, a joint session was organized with one or two participants from each organization to discuss the most important themes named by the participants. This session was also recorded, and it provided additional insight about the interview data. This section presents the main limitations of the study through an assessment of its internal, construct, and external validity. Internal validity needs to be assessed when causal relationships are studied [15]. The purpose of the study was not to establish causal relationships. Some of our findings have a causal nature and it is possible that unknown affecting factors exist although we had thorough understanding of the cases. Construct validity is concerned with the design of a study and whether the studied artifacts really represent what the researchers had in mind [15]. We examined more than one company, so triangulation of companies is present. We had limited triangulation of the data collection methods. We relied mostly on the two interviews performed at each organization. Additionally, we benefitted from our previous experiences with the companies [9], and from the informal A little weak communication that was used to clarify the information obtained during the interviews. Due to the research collaboration, we examined the cases and their contexts over a longer period, which strengthens the results Taken from a Colleague: Validity (2) End result All of the research was done by two people, which limited researcher bias to some degree. However, as the two people were working closely together rather than independently the researcher bias is not completely eliminated. External validity is concerned with whether it is possible to generalize the results [15]. Generalizing results is difficult in software engineering because of the effect of context [6, 7, 13]. As a case study offers a limited possibility of generalization, our case selection has undoubtedly affected the results. As discussed in Section III.B, two vendors also had divisions providing COTS products, but they were not selected for the study as they had no problems with deliveries. We welcome papers that specifically describe case studies such as this one The approach is well described and we compliment the authors on the inclusion of a validity defense The research method is well described and although I do not subscribe to the generalizability of the cases, the validity defense makes a good attempt to defend it. (from three different accept notifications) Although this limits one s ability to generalize the findings, we can also interpret this as a tentative result indicating the impact of the software s operational environment to the deployment process. It seems very intuitive to think that standard COTS products result in fewer deployment-related difficulties compared to products closer to Enterprise systems

10 Summary Case Study: an intensive, detailed description and analysis of a single project or program in the context of its environment. You should be able to create a chain of evidence Case study research questions Protocol -> How do you answer your questions? Questions? Case study database Case study report A correct validity defense of your research by defending Construct validity Internal validity External validity Repeatability slinger@cs.uu.nl Further reading: Robert K Yin. Case Study Research - Design and Methods. SAGE Publications, 3rd edition, 192 pages, London, England,