Selecting a research method

Transcription

1 Selecting a research method Tomi Männistö Overview Theme Maturity of research (on a particular topic) and its reflection on appropriate method Validity level of research evidence Part I Story from another world Part II Mary Shaw s viewpoints on selecting research method 1

2 Story of Research Methods Case Nutritional Epidemiology Relating research methods Analogy from nutritional epidemiology Why such a case? Measuring (food consumption) inherently problematic More mature in terms of research methodology than SE A range of methods available Easy to explain General research question Association between a food (or nutrient, consumption pattern, nutrition related facto) and disease (risk) E.g., cholesterol vs. coronary hearth disease, or coffee vs. diabetes (protective?) 2

3 Basic research methods Pooled analysis Combine data from multiple studies Meta analysis Analyse the reported results of multiple studies Experiment on humans Control and measure everything the subjects eat Longitudinal study, follow-up, cohort Follow subjects over time Case-control study Identify cases that have developed a disease, compare past exposure to the selected factor with controls that do not have the disease Cross sectional study, biographical studies, trend analysis Prevalence of disease, fat consumption vs. heart disease in different countries Experiment on animals In vitro studies Measuring food consumption Food records (diary) Write down everything your eat 24 recall, 7-day food records, 14-day food records Food Frequency Questionnaire (FFQ) E.g., on average, how much and frequently you have eaten potatoes during the last week, year A: daily 3 potatoes at the time or once a week 3

4 Validating a method Using less expensive method Validation against a more accurate one with small sample Reliability FFQ1 vs. FFQ2 (3 months in between) Validity FFQ1 vs. Diary (min. 7-days) Inherent, systematic errors of methods must not correlate Interpretation of results = know your method! Possibly correction by statistical means Concluding (discussion) points Hierarchy of methods Increasing use of resources, cost, time, people Less matured hypotheses built or tested with lower method Using too heavy method for vague hypothesis is waste of resources The higher the better not that simple!!! Different subject of study in vitro vs. rats vs. (small group of )humans vs. populations Know your method Analogies to SE? 4

5 SA / SE Research (Mary Outline Basic points Types of research results Types of research results / strategies Types of research validation Possible plans 5

6 Basic points Ideas mature over time They grow from qualitative and empirical understanding to precise and quantitative Research questions are of different kinds Kinds of interesting questions change as ideas mature Research strategies also vary They should be selected to match the research question Building blocks for research 6

7 Types of research questions Feasibility Characterization Method / Means Generalisation Discrimination / Selection Is it possible to do X? Is it possible to describe relations among components of a software system? Is it possible to automatically generate code from an architectural specification? Does X exist, and what is it? Types of research questions Feasibility Characterization Method / Means Generalisation Discrimination / Selection What are the characteristics of X? What, exactly, do we means by X? What are the varieties of X, and how are they related? What is the important information to share and hide about a component? What are the important concepts for modeling software architectures? 7

8 Types of research questions Feasibility Characterization Method / Means Generalisation Discrimination / Selection How can we do X? What is a better way to do X? How can we automate doing X? How can we incorporate abstract data types in a programming language? How can we exploit domain knowledge to improve software development? Types of research questions Feasibility Characterization Method / Means Generalisation Discrimination / Selection Is X always true of Y? Given X, what will Y be? What is a formal relation between the specification of an abstract data type and its implementation? What patterns capture and explain a significant set of architectural constructs? 8

9 Types of research questions Feasibility Characterization Method / Means Generalisation Discrimination / Selection How do I decide whether X or Y? What organization should I choose for a user interface component? How can a designer tradeoff choices among architectural alternatives? Building blocks for research 9

10 Strategy / results Qualitative & descriptive models Report interesting observations Generalize from (real-life) examples Structure problem area Ask good questions Early architectural models Architectural patterns Techniques Invent new ways to do some tasks, including implementation techniques Develop ways to select from alternatives Domain-specific software architectures UML to support object-oriented design Strategy / results (2) System Embody result in a system, using the system both for insight and as carrier of result Architecture description languages Empirical models Develop empirical predictive models from observed data Communication metrics as indicator on project complexity Analytic models Develop structural models that permit formal analysis Formal specification of higher level architecture for simulation 10

11 Building blocks for research Validation Persuasion I thought hard about this, and I believe... Early architectural models Implementation Here is a prototype of a system that... Early architecture description languages Evaluation Given these criteria, the object rates as... Taxonomies; performance improvement 11

12 Validation (2) Analysis Given the facts, here are consequences... Many predictive models Formal model Rigorous verification and proof Empirical model Data on use in controlled situation User interface structures Experience Report on use in practice Case studies Qualitative model Narrative Architectural patterns Decision criteria Comparison of systems in actual use Domain-specific architectures Empirical model Data, usually statistical, on practice Communication and project complexity A common plan Method - System - Evaluation Can X be done better? - Build Y - Measure Y, Compare to X Common, but bad Method - Technique - Persuasion Can X be done better? - Devise a technique - "Look it works!!" Other good plan 1 Feasibility - System - Persuasion Can X be done at all? - Build Y that does X - "Look it works!!" Other good plan 2 Generalisation - Analytic model - Analysis Is X always true of Y? - Formally model Y, prove X - Check proof? Sometimes breakthrough sometimes nonsense Characterization - Qualitative model -Persuasion Change basic assumptions - Formulate new approach - Argue carefully on merits Possible plans 12