Engineering Science & VALIDITY Today we will look at: 1. Scientific foundations 2. Validation 3. Reliability 4. Logic 5. Industrial validation 6. MSc relevance February 24, 2014 1
The linked image cannot be displayed. The file may have been moved, renamed, or deleted. Verify that the link points to the correct file and location. 0 Scientific Foundations Research can be defined as the scientific search for knowledge, or as any systematic investigation, to establish novel facts, solve new or existing problems, prove new ideas, or develop new theories, usually using a scientific method. The primary purpose for basic research is discovering, interpreting, and the development of methods and systems for the advancement of human knowledge on a wide variety of scientific matters of our world and the universe, while applied research tends to solve existing or new problems. Validation is proving that the research is indeed true and based on strong evidence! February 24, 2014 2 Basrelief sculpture "Research holding the torch of knowledge" (1896) by Olin Levi Warner. Thomas Jefferson Building, Washington, D.C
Research defined As per the Merriam-Webster Online Dictionary, the word research is derived from the Middle French "recerche", which means "to go about seeking", the term itself being derived from the Old French term "recerchier" a compound word from "re-" + "cerchier", or "sercher", meaning 'search'. The earliest recorded use of the term was in 1577. February 24, 2014 3
Other definitions A broad definition of research is given by Martin Shuttleworth - "In the broadest sense of the word, the definition of research includes any gathering of data, information and facts for the advancement of knowledge." Another definition of research is given by Creswell who states - "Research is a process of steps used to collect and analyze information to increase our understanding of a topic or issue". It consists of three steps: Pose a question, collect data to answer the question, and present an answer to the question. February 24, 2014 4
So validation completes the scientific method Observations and Formation of the topic: Consists of the subject area of ones interest and following that subject area to conduct subject related research. The subject area should not be randomly chosen since it requires reading a vast amount of literature on the topic to determine the gap in the literature the researcher intends to narrow. A keen interest in the chosen subject area is advisable. The research will have to be justified by linking its importance to already existing knowledge about the topic. Hypothesis: A testable prediction which designates the relationship between two or more variables. Conceptual definition: Description of a concept by relating it to other concepts. Operational definition: Details in regards to defining the variables and how they will be measured/assessed in the study. Gathering of data: Consists of identifying a population and selecting samples, gathering information from and/or about these samples by using specific research instruments. The instruments used for data collection must be valid and reliable. Analysis of data: Involves breaking down the individual pieces of data in order to draw conclusions about it. Data Interpretation: This can be represented through tables, figures and pictures, and then described in words. Validation & test, revising of hypothesis Conclusion, reiteration if necessary February 24, 2014 5
1 Introducing Validity February 24, 2014 6
Validation defined The quality of various conclusions you might reach based on a research project and the best available approximation to the truth of a given proposition, inference or conclusion February 24, 2014 7
Two realms involved in research: The land of theory: what the cause is believed to be and what the effect identified is (affecting something and measuring it) The land of observation: observing effects as the outcome of a research program testing hypotheses Operationalisation is the act of translating a theoretical construct into its manifestation (taking the idea and describe it as a series of operations and procedures) February 24, 2014 8
Realms and components of research: February 24, 2014 9
Four validity types: External validity is primarily concerned with the range of theory Construct validity emphasises the linkages between theory and observation Conclusion validity and Internal validity refer to the land of observation February 24, 2014 10
How valid is the theory? Conclusion validity: in a research study, is there a relationship between two variables identified? Internal validity: if so is the relationship a causal one? Construct validity: given a causal relationship, did the operationalisation of the theoretical constructs reflect the ideal typically were you able to measure what you wanted to capture? External validity: given the causal relation between the constructs of cause and effect, can you generalise this effect to other persons, cases, times or places etc? February 24, 2014 11
Validity staircase: types of validity: February 24, 2014 12
1.2 Conclusion Validity February 24, 2014 13
Conclusion validity Conclusion validity is the degree to which conclusions you reach about relationships in your data are reasonable Invariably involving statistical analysis: a branch of mathematics dealing with the collection, analysis, interpretation, and presentation of quantitative data February 24, 2014 14
Threats to Conclusion Validity Threats: any factor that can lead you to reach an incorrect conclusion about a relationship in your observations Either: Concluding there is a relationship when there is not (seeing things that are not there) or Concluding there is not a relationship when there is one (missing the needle in the haystack)! February 24, 2014 15
Improving Conclusion Validity Good statistical power: the probability of correctly concluding that there is a treatment/program effect in your data 80% chance of finding a statistical relationship when there is one (0.8)! Good reliability reducing noise! Good implementation training operators, standardising administration and result measuring protocols February 24, 2014 16
1.3 Internal Validity February 24, 2014 17
Internal Validity Research design is intrinsically linked to internal validity and is constructed to minimise the threat to internal validity! The internal validity defines the quality of the research design! Internal validity: The approximate truth of inferences regarding cause-effect or causal relationships can the observed changes be attributed to your program or intervention (the cause) and not to other possible causes (alternative explanations for the outcome) but doesn t care about whether the cause-effect relationship would be found in any other context outside the study February 24, 2014 18
Establishing Cause and Effect Generally you must meet three criteria before you can say that you have evidence for a causal relationship: Temporal precedence Covariance of the cause and effect No plausible alternative explanation February 24, 2014 19
Temporal precedence One criterion for establishing a causal relationship that holds that the cause must occur before the fact Not always clear: Does inflation cause unemployment increased inflation could cause unemployment but increased employment could cause inflation through supply and demand February 24, 2014 20
Covariance of the cause and effect A criterion for establishing a causal relationship that holds that the cause and effect must be related or covary If X then Y; If not X then not Y If program then outcome, if not program then not outcome But: doesn t provide evidence that the program caused the outcome could be some other associated cause!!! February 24, 2014 21
No plausible alternative explanation Any other cause that can bring about an effect that is different from your hypothetical or manipulated cause Just because you show a relationship doesn t mean it is a causal one! Core to research design in being central to the internal validity To argue that you have internal validity that you have shown there s a causal relationship you have to rule out the plausible alternative explanations Use of a control group subject to circumstance but not your program February 24, 2014 22
1.4 Construct Validity February 24, 2014 23
Idea of construct validity Construct validity is degree to which inferences can legitimately be made from the operationalizations in your study to the theoretical constructs on which those operationalizations were based! February 24, 2014 24
Convergent validity correlations Convergent validity is showing that measures that should be related are related! February 24, 2014 25
Divergent validity correlations Divergent validity is showing that measures that should not be related are in reality not related! February 24, 2014 26
1.5 External Validity February 24, 2014 27
External Validity External validity: degree to which conclusions hold true for others in other locations Sampling model: take population then draw representative sample Proximal similarity model: different contexts that you want to then generalise to and similarity to your study according to: gradient of similarity (relative to your study!) February 24, 2014 28
Sampling model for external validity February 24, 2014 29
Proximal similarity model for external validity February 24, 2014 30
Ensuring external validity Threats to external validity: what could be wrong in generalising your conclusions (people, place, time, experiment, etc) Improving external validity: good sampling from population, good participation, understanding the gradient of similarity, prove through test that your generalisations do hold true with extended testing (validation) February 24, 2014 31
3 Reliability February 24, 2014 32
Reliability and true score theory Reliability is the degree to which a measure is consistent or dependable; the degree to which it would give you the same result over and over again, assuming the underlying phenomenon is not changing True score theory maintains that every measurement is an additive composite of two components: the true ability of the respondent (or experiment etc) and the random error February 24, 2014 33
Random error Random error adds variability to a distribution but does not affect the central tendency or the average (randomly affecting measurement of the variable across the sample) February 24, 2014 34
Systematic error Systematic error affects the central tendency of a distribution (affecting measurement of the variable across the sample) February 24, 2014 35
Reliability and true score theory Even if scores are the same, the error will differ for different people, time and places! Reliability is the ratio of to true level of the measure to that found in the study! February 24, 2014 36
Types of reliability Test-retest reliability (two different occasions!) Parallel-forms reliability (same construct different test split half reliability!) Inter-observer reliability (people are notorious for inconsistency!) February 24, 2014 37
The shooting target metaphor Reliability and validity are closely related to each other! February 24, 2014 38
4. Logic! February 24, 2014 39
Is this valid? All men are mortal. Socrates is mortal. Therefore, Socrates is a man. February 24, 2014 40
No In this case, the conclusion does not follow inescapably from the premises. All men are mortal, but not all mortals are men. Every living creature is mortal; therefore, even though both premises are true and the conclusion happens to be true in this instance, the argument is invalid. It could have been Mrs. Socrates! February 24, 2014 41
Is this the following well-known syllogism valid? All men are mortal. Socrates is a man. Therefore, Socrates is mortal. February 24, 2014 42
Yes What makes this a valid argument is not that it has true premises and a true conclusion, but the logical necessity of the conclusion, given the two premises. February 24, 2014 43
Is this valid? All cups are green. Socrates is a cup. Therefore, Socrates is green. February 24, 2014 44
Yes! What makes this a valid argument is not that it has true premises and a true conclusion, but the logical necessity of the conclusion, given the two premises. The argument would be just as valid were the premises and conclusion false. The previous argument is of the same logical form but with false premises and a false conclusion, and yet it is equally valid! February 24, 2014 45
Abstracting! A standard view is that whether an argument is valid is a matter of the argument s logical form. Many techniques are employed by logicians to represent an argument s logical form. A simple example, applied to two of the above illustrations, is the following: Let the letters P, Q, and S stand, respectively, for the set of men, the set of mortals, and Socrates. Using these symbols, the first argument may be abbreviated as: All P are Q. S is a P. Therefore, S is a Q. Similarly, the third argument becomes: All P are Q. S is a Q. Therefore, S is a P. An argument is formally valid if its form is one such that for each interpretation under which the premises are all true, the conclusion is also true. As already seen, the interpretation given above does cause the second argument form to have true premises and false conclusion, hence demonstrating its invalidity. February 24, 2014 46
Validity & soundness Validity of deduction is not affected by the truth of the premise or the truth of the conclusion. The following deduction is perfectly valid: All fire-breathing rabbits live on Mars All humans are fire-breathing rabbits Therefore all humans live on Mars The problem with the argument is that it is not sound. In order for a deductive argument to be sound, the deduction must be valid and all the premises true. February 24, 2014 47
5. An industrial perspective! February 24, 2014 48
Industry validation It is sometimes said that validation can be expressed by the query "Are you building the right thing?" and verification by "Are you building it right?" "Building the right thing" refers back to the user's needs, while "building it right" checks that the specifications are correctly implemented by the system. In some contexts, it is required to have written requirements for both as well as formal procedures or protocols for determining compliance. February 24, 2014 49
Industry validation: most tested attributes Selectivity/specificity Accuracy Precision/Repeatability Reproducibility Limit of detection - especially for trace elements Limit of quantification Curve fitting and its range System suitability February 24, 2014 50
Industry Verification and Validation Verification is a quality control process that is used to evaluate whether a product, service, or system complies with regulations, specifications, or conditions imposed at the start of a development phase. Verification can be in development, scale-up, or production. This is often an internal process. Validation is a quality assurance process of establishing evidence that provides a high degree of assurance that a product, service, or system accomplishes its intended requirements. This often involves acceptance of fitness for purpose with end users and other product stakeholders. This is often an external process. February 24, 2014 51
The famous industry V diagram February 24, 2014 52
Design verification test (DVT) Design verification test (DVT) is an intensive testing program which is performed to deliver objective, comprehensive testing verifying all product specifications, interface standards, OEM requirements, and diagnostic commands. It consists of a number of areas of testing: February 24, 2014 53
Areas of DVT testing Functional testing (including usability) Performance testing Climatic testing Reliability testing Environmental test Mechanical test MTBF prediction Compliance and regulatory testing EMC test and certification Safety certification February 24, 2014 54
Six Sigma Validation Example February 24, 2014 IQ - Installation Qualification OQ - Operational Qualification PQ - Performance Qualification 55
EUROCONTROL Validation Example mid 90s 2000 2001 2002 2003 2004 Recognition of lack of Business Case support to decision making EUROCONTROL makes the Validation Data Repository available (June) MAEVA provides first version of the Validation Guide Handbook (June) MAEVA project completed (April) CAATS project started (April) First meeting of the Joint Programme Board (JPB) (Oct) 2005 JPB endorses E-OCVM for collaborative projects for EC and EUROCONTROL (Jan) Validation Forum Supervisory Board set-up (May) February 24, 2014 2006 2007 2008 Release of E-OCVM v 1.0 (June) First meeting of Validation Forum Supervisory Board (Nov) CAATS project completed (March) CAATS II project started (Nov) Release of E-OCVM v 2.0 Episode 3 project started (April) 56 2009 CAATS II project completed (Nov) Episode 3 project completed (Dec) 2010 Release of E-OCVM v 3.0
V0 ATM Needs V1 Scope V2 Feasibility V3 Integration K ey Identification and prioritisation of ATM problems and performance targets Select among potential concept & supporting enabler alternatives & options Select among FEASIBLE concept & supporting enabler alternatives & options Handover from R&D to the Industrialisation process D ec i s i o n s Go/No-Go for call for projects/solutions Allocate resources & plan - based on dependencies, prioritisation, maturity urgency, budget, etc. Go/No-Go for V2. Update resources & plan - based on dependencies, prioritisation, maturity vs. urgency, budget Go/No-Go for V3 Initial Go/No-Go for deployment of the preferred alternative & options February 24, 2014 57
Concurrent activities: ATM Needs Scope Feasibility Pre-industrial development & integration Industrialisation Deployment Operations Decommissioning V0 V1 V2 V3 V4 V5 V6 V7 Concept Validation (E-OCVM) Requirements development Concept development Technical development and Verification Integration Examples of other key ATM system development activities February 24, 2014 58
Interdisciplinary activities: Business Case Safety Case Human Factors Case Environment Case Standards Case Regulatory Case Pre-industrial ATM Needs Scope Feasibility development & Industrialisation Deployment Operations Decommissioning integration V0 V1 V2 V3 V4 V5 V6 V7 Gather and assess ATM Performance needs Scope Operational Concept and develop Validation Plans Iteratively develop Build, consolidate and evaluate and test concept E-OCVM Scope Industrialisation and Approval Installation and roll-out Implementation Removal and replacement February 24, 2014 59
6. An LR MSc thesis perspective! February 24, 2014 60
6. So what does all this mean to a TUD aerospace student? Is the basic hypothesis of your thesis research a valid formulation, i.e. relative to your fundamental statement where you identify a potentially causal link between variables & parameters and an outcome? Be very careful in formalizing your research hypotheses! Can your research goal be validated and proven to be true? Plan your research design to accommodate the required validation, not just explore and implement! How high will be the quality of the validation in your thesis research? Design a validation exercise into your thesis work! February 24, 2014 61
Practically, MSc design for validation through: Having a clear research hypothesis to be tested positive, negative or null! Having a clear way to test it, how could you prove it beyond all shadow of doubt? Design your research program (your test!) with a realistic view on the time scales, data acquisition, analysis quality, and sensitivities. Also validate sub-components of the work as well as the overall outcome. Don t leave it to the last minute; 2 weeks at the end or even the last month! In the final examination, being able to argue with scientific evidence that your findings are original, proven, and relevant! February 24, 2014 62
So can you work towards a clear research goal that can be validated? February 24, 2014 63
The ultimate proof An engineering relevant journal paper! Most rejections (75%) of academic journal submissions are due to either insignificant impact or poor validation! One can argue : Academia is about the pursuit and development of new knowledge A piece of original TUD MSc work that is validated is new knowledge and needs to be disseminated (published) in the most reputable journals. Therefore journal publication is the ultimate success of an MSc project! And lets not forget, industry want to have confidence in the reliability of the results from university models and tools! And eventual industrial application should lead to value creation (value engineering) and the improvement of societal quality of life! February 24, 2014 64
Finally check out some helpful sites such as: http://www.boincwiki.info/a_simple_example_of_the_validation_proces s http://en.wikipedia.org/wiki/crossvalidation_(statistics) February 24, 2014 65