POD data collection & analysis Tools for beginners

Transcription

1 4th European-American Workshop on Reliability of NDE June 2009 Reliability for NDT Tutorial: POD Basic POD data collection & analysis Tools for beginners

2 Introduction Round: - How are you involved with NDE reliability in your present role? - What do you hope to accomplish in the next 6 months with the tools learned here?

3 Goals: Comfortable using typical POD software and presentation tools Can produce valid results Aware of sources of variation and mitigation steps

4 Basics Roadmap POD data collection & analysis Tools for beginners - Before the start: Background on NDE Reliability Measurement Practice - Vetting and assessing existing databases of POD capability - Completing experimental gap analysis - Calibrations & transfer functions - Protocols for partial/full capability demonstrations - Quality checking the data - Selecting a model - Understanding capability relationships behind POD curves - Understanding & addressing experimental variation - Substantiating and presenting the results Goal: 1/3 Orientation & Discussion, 2/3 Performing POD calculations

5 Basics Roadmap POD data collection & analysis Tools for beginners - Before the start: Background on NDE Reliability Measurement Practice - Vetting and assessing existing databases of POD capability - Completing experimental gap analysis - Calibrations & transfer functions - Protocols for partial/full capability demonstrations - Quality checking the data - Selecting a model - Understanding capability relationships behind POD curves - Understanding & addressing experimental variation - Substantiating and presenting the results

6 NDT Reliability - Background Degree that an NDT system is capable of achieving its purpose regarding detection, characterization and false calls American European Workshop on NDT reliability 99 Quantitative measure of the efficiency of the NDT procedure in finding flaws of specific type and size Metals Handbook

7 Quantification Ideally Cracks larger than a certain size can be detected Reality There is a probability of detection for every crack No Detect Probability of Detection Detect Defect Size

8 NDT Reliability Measurement POD - Probability of Detection - Is it an adequate representation? POFA - Probability of False Alarm - Equally important from economic consideration ROC - POD vs. POFA - A measure of reliability Coefficient of Contingency - A measure of individual performance

9 POD Curve (Safety) % Confidence bound POD 90/95 Crack size Defect Size POD = F(Finds)

10 ROC Curve (Economics) > 80% Finds < 20% False calls PO True Call ROC = F(Finds, False Calls) PO False Call

11 Coefficient of Contingency Flawed Unflawed Marked Finds (TP) False Calls (FP) Not Marked Misses (FN) True no-calls (TN) Coeff of Contingency = F (TP, TN, FP, FN)

12 Challenge Human Factors Variables Identification Control Quantification Operator-Equipment-Environment interaction

13 What Do We Need to Understand? NDT is not viewed as a friend of production The program is in operational interest Identify and eliminate deficiencies in NDT system Ultimately operate safer, cheaper, longer

14 Popular Characteristics of POD POD is Expensive Certified inspectors do not need POD Experienced and high salaried inspectors have better POD 90/95 Crack size information is adequate Imp to find small flaws

15 How is the product of NDE measurement Engineering used?

16 Damage Tolerance Concept Damage size Estimated tolerable damage size Poorer Assumed than detectable assumed Damage size Better than assumed Economic? Safe? Inspection Interval Service Opportunity for damage detection

17 Damage Tolerance Concept Tolerable damage Damage size Detectable damage Assessed reliability Improved Reliability Service Inspection Interval Opportunity for damage detection

18 First things first! Relation of system response to POD curve Regression equation from A-hat.exe program L. Schaefer NIST 1998 A-E NDE Reliability 50% POD point for an 83 unit decision threshold 50% POD point for an 129 unit decision threshold

19 Roadmap Basics POD data collection & analysis Tools for beginners - Before the start: Background on NDE Reliability Measurement Practice - Vetting and assessing existing databases of POD capability - Completing experimental gap analysis - Calibrations & transfer functions - Protocols for partial/full capability demonstrations - Quality checking the data - Selecting a model - Understanding capability relationships behind POD curves - Understanding & addressing experimental variation - Substantiating and presenting the results

20 30 years of NDE Reliability studies Step 1: Mimimize cost of knowledge acquisition by reviewing literature first! NTIAC Karta MAPOD FAA-ETC Abundant peer reviewed NASA, DoD, EPRI data

21 Working with data sets: Lets try a few! Exercises 1 Review POD Rev 3 user instructions Examine and plot in excel the data located under the FPI directory (26 Perfect, L3NQ) of a-hat vs a data Prepare a separate sheet with the data transformed into hit/miss (0, 1), based on hit = 30 Conduct a-hat vs a and pass/fail analyses with the provided threshold information Compare & discuss results Caution for Excel treatment of commas vs point! Use whole numbers

22 Lets try a few! Exercises 2 Examine and plot in excel the data located under the ET directory (26Parent) of a-hat vs a data Prepare a separate sheet with the data transformed into hit/miss (0, 1) where hit = 100 Conduct a-hat vs a and pass/fail analyses with the provided threshold information Discuss results

23 Lets try a few! Exercises 3 Examine and plot in excel the data located under the UT directory (UT270) of a-hat vs a data Conduct a-hat vs a with the provided threshold information Present & Discuss results More or continue?

24 Roadmap Basics POD data collection & analysis Tools for beginners - Before the start: Background on NDE Reliability Measurement Practice - Vetting and assessing existing databases of POD capability - Completing experimental gap analysis - Calibrations & transfer functions - Protocols for partial/full capability demonstrations - Quality checking the data - Selecting a model - Understanding capability relationships behind POD curves - Understanding & addressing experimental variation - Substantiating and presenting the results

25 Gap Analysis We dove into the deep end, now lets reflect: What needs to be done now to assure the results are correct? Did we obey our process? Substantiating and presenting the results: Understanding & addressing experimental variation Calibrations & transfer functions Protocols for partial/full capability demonstrations Quality checking the data Selecting a model

26 Objective: Identify and Eliminate Deficiencies NDT system capability Knowledge on reasons for the gap? Recommend actions to bridge the gap NDT system performance

27 Approach to Inspection Reliability Advanced Reliability Program Improved Reliability Recommended Actions Knowledge - a) System Capability, b) Improvement Avenues Designed Experiment Classic POD Program Reliability Information POD Analysis Data base

28 What Constitutes an NDT System? Environment Human Application Equipment

29 What Factors Influence the Most? Human Factors Application condition, access, Equipment sensitivity, resolution, complexity, Process, Materials, Interactions

30 Human Factors still a challenge Factors that impact inspector s discrimination and decision-making ability Organizational Physical Mental Training and skill level is a major factor Inspections with predictable outcome Routine and monotonous

31 Roadmap Basics POD data collection & analysis Tools for beginners - Before the start: Background on NDE Reliability Measurement Practice - Vetting and applying existing databases of POD capability - Completing experimental gap analysis - Calibrations & transfer functions - Protocols for partial/full capability demonstrations - Quality checking the data - Selecting a model - Understanding capability relationships behind POD curves - Understanding & addressing experimental variation - Substantiating and presenting the results

32 What Constitutes an NDT System? Environment Human Application Equipment

33 Typical NDT Assessment Program Creation of specimens with defects Visit to an NDT facility Identification of a sample of inspectors Conduct of NDT on set of specimens Acquisition of inspection data Data analysis and POD plots

34 NDT Assessment Program Elements and sources of variation Facility Sampling Inspector Sampling Specimens Scheduling Inspections Data Acquisition Data Analysis Human factors

35 Specimens Ideally, real parts with real cracks Typically, synthetic parts or a combination of real and synthetic parts Configuration as close to critical inspections as possible Presentation as close to real situation as possible Special care in handling and maintenance

36 Specimens Multiple identical specimens Mounted on framework (racks) with quick interchangeability feature Multiple inspection sites per specimen Uniquely numbered for tracking Specimen inspection guideline similar to written procedures Routine surface cleaning process without damage

37 Specimen Defects defects per set Most flaws in the zone of increasing POD Preferred 10-90% Typically 1-99% (Hard to judge) Preferred size distribution linear on log a scale Flawed : Unflawed site :: 1 : Well characterized initially and regularly For details refer to MIL-HDBK-1823

38 Specimen Fabrication Raw Specimen Flaw growth Final shape and size Characterize and mark

39 Cracked Metal Specimens Raw Specimen Specimen EDM Crack Surface crack Grip Area Margin Raw Specimen Specimen Specimen Corner crack Through Crack EDM Raw Specimen EDM Specimen Raw Specimen

40 Corroded Metal Specimens ASTM Standards Grip Area Margin Margin Raw Specimen Specimen Machined Corrosion SpecimenRaw Specimen Specimen Painted Specimen Exposed Environment

41 Response Matrix Flaw No Flaw Marked Find False Call Not marked Miss True no-call Marked Presence or absence Size quantified

42 Signal/noise Discrimination Signal amplitude Probability density function Noise Decision Signal Misses False Calls

43 Signal/noise Discrimination Noise Signal Poor Good Poor Noise Signal Poor process/setup Poor Discrimination Poor reliability Good procedure, equipment, Inspector dependent reliability

44 Data Analysis Demonstration of capability at one crack length Determination of POD function through single inspection of cracks covering a range of lengths Binomial Distribution Theory (Grouping) Estimation of POD function and confidence bounds through multiple inspections of cracks covering a range of lengths Regression Analysis (Curve fitting)

45 Data Analysis Two Steps Generate a point estimate of detection probability for various crack lengths over a range of interest Fit an appropriate curve that offers minimum deviation or maximum likelihood to the scattered data

46 Log-odds Model = = = c a x c a p y c a e c a ln ; ˆ ln ln ; 1 ) POD( β α

47 a vs. a Analysis Consider a lognormal scatter in indicated crack length for various cracks lengths POD is the probability of indicated crack length exceeding the threshold of detection Requires quantification of signal leading to detect call Indicated crack length (a) v Threshold True crack length (a) POD True crack length (a)

48 Experimental quality assessments Example One Human Factors quantification in Eddy Current POD Study

49 Demonstration Study - Introduction Protocol for Human Factors Quantification Field lab visited in Sept 2000 L8 with 3 DOE variables Job Type (Part Time vs Full Time) Physical comfort (Comfortable vs Uncomfortable) Speed of inspection (Normal vs Accelerated)

50 Demonstration Study - Plan Eddy Current specimen set (Mfg at SwRI) 15 specimens, 8 sites/specimen 30 cracks, mil 15 inspectors (8 full time, 7 part time) 4 rounds per inspector Uncomfortable/comfortable, normal/fast 58 inspections on 120 sites

51 Demonstration Study Specimens

52 Demonstration Study - Inspections

53 Demonstration Study POD Curves 1.0 Effect Effect of Schedule of Physical Job Type Comfort on POD on POD 6 Uncomfortable Part Normal time inspectors, speed, Posture, 2924 inspections 28 inspections 9 Full Comfortable time Accelerated, inspectors, Posture, 2934 inspections 30 inspections POD Uncomfortable Part Fast time 95% 95% Comfortable Full Normal Time 95% 95% Crack Size Size (mils)

54 Demonstration Study DOE Analysis DOE Analysis Normalized 90% POD Crack size GM J P JP S JS PS E

55 Example Study Effect of Physical and Schedule comfort Individual Performance Coefficient Day-2 Day-1 Uncomf-Fast Uncomf-Normal Comf-Fast Comf-Normal 0 Inspector

56 Example Two Considerations in FPI - POD

57 FPI - Reliability Formula Elements IC - Chemistry, fluid mechanics AP - Material, surface condition, location HF - Contrast, spatial perception

58 FPI - Process parameters

59 FPI - Process parameters

60 FPI - A-hat vs a, or Pass/Fail? Pass/fail model-program considerations Process and behavior must conform to model assumptions. or no result/non-sense result Asymptotic signal to noise A-hat vs. a - Preferred Given - Sufficient quantity of data - as few as 20 points - Measurable strength of response At minimum, mean performance can be calculated

61 FPI - Calculations; misses and false call management Effect of decision threshold changes; 5-30mils Penetrant POD performance as a function of decision threshold 1.20E E E-01 POD 6.00E E E E E E E E E E E E E-01 flaw length (in.)

62 FPI - Source data from flat plates ra ra ra ra ra ra ra ra ra ra ra ra ra ra ra ra ra ra ra ra ra ra ra ra ra ra ra ra ra ra ra ra ra ra ra ra ra ra ra ra ra ra ra ra ra ra ra ra ra ra ra ra ra ra ra

63 AP - Isolating chemical parameters... Influences of FPI parameters using flat panels A-hat 90/95 CIF(nde) length (inches) L 3 W-W A-NQ L3 W-W AB-NQ L 4 PE AB-NQ TL-KN L4 WW AB-NQ L3 WW DP Linear (L3 WW DP) Linear (L3 W-W AB-NQ) Linear (L 3 W-W A-NQ) Linear (L4 WW AB-NQ) Set "A" of NASA-SSME panels used for all exams. Panels include 56 cracks on 15 4"x15" panels (both sides). Size.016" to.118 (.361 extraneous, non-verified flaw) Inspector -& Group performance...

64 FPI - Group performance expectations Influences of FPI parameters using flat panels 0.07 A-hat 90/95 CIF(nde) length (inches) Inspector Group

65 Example Three Laser Methods - POD Shearography

66 Application - Bonded Structures, types Edge Assembly Ramp Support Structure Thrust Ramp Engine End Close-out Honeycomb is A-286,.032 facesheets over 1.4 thick core,.5x.2~ cell size. Core is perforated. Base Close-out Traditionally acoustic inspections UT, resonance, tap facesheet liner Lasers can sense, map bond related displacement

67 Shearography - What can we measure?

68 Shearography - A-hat vs a analysis Shearography response area (sq.in.) st Cycle Braze A-hat Scatter (168 responses) Area (sq. in.) -.2" min dim fwd>aft POD/MTD/Prod. Data POD from X-33-LS-01 Base Area Width Length Flaw ID 1.25x Linear (POD/MTD/Prod. Data) 2.25x x x x x x x x x x x x x x x x x x x x x x y = 1.03x 24.25x R 2 = POD 1st Cycle Braze Liner/Closeout Shearography POD (Valid for unbonds.2" and larger) Unbond Area (.2"x.2" Decision threshold) 90/95=.104 sq. in.

69 Comparing the old vs. the challenger We can assess the discrete reliability as a function of decision threshold

70 Ultrasonic A-hat vs a for HC panel ap ap ap ap ap ap ap ap ap ap ap ap ap ap ap at at at at at at at at at at at at at at at at at at at at ab ab ab ab ab ab ab ab ab ab bp bp bp bp bp bp bp bp bp bp bp bp bp bp bp Measured Unbond Dimension (in.) Threshold: 5x5 pixels >= 6db Ultrasonic scatter data for.016"skin,.1875" cell panel Programmed Unbond Dimension (inches)

71 Shearography A-hat vs a for HC panel ap ap ap ap ap ap ap ap ap ap ap ap ap ap ap at at at at at at at at at at at at at at at at at at at at ab ab ab ab ab ab ab ab ab ab bp bp bp bp bp bp bp bp bp bp bp bp bp bp bp Measured unbond (in.) Shearography response scatter for.016" skin,.1875 cell panels Programmed unbond dimension (inches)

72 Example Four Pressure Vessel POD Ultrasonic & Eddy Current

73 PV-POD Using transfer functions to fill gaps Often we are asked to develop POD But we can not create perfect knowledge Exact material Exact geometry The precise flaws expected in the design - All orientations - All morphologies How can we approximate what we do not know?

74 Pressure vessel POD - Transfer functions Target application; 120mm welded vessel Internals are expensive, can t afford false calls Thin - 1mm wall Situation - Resources to assess with LCF cracks in flat plate - welded to spec - Non-welded plate 5mm Automated UT & EC - Acquisition & Analysis EDM artifacts for all critical locations Must estimate differences from lab to field

75 Transfer process Compare family of EDMs across thicknesses Compare family of EDMs across geometries & PM vs weld

76 PV - POD; Results ET PV ET Scatter of Mean Responses for 26 Crack Sample PV ET POD for Battery Parent Metal (from 26parent.pod) 1.5 A/D POD Crack Length (in.) Flaw Length (in.)

77 PV - POD; Results UT Mean ultrasonic response versus length Amplitude (mv) Crack Length (inches) Discipline in calibration will assure estimate holds in practice!!

78 Example Five Radiographic POD considerations

79 Radiography - HF dependant Despite advances in image processing most applications rely on human interpretation PM Weld Detection targets include much beyond simple cracks of length and depth Pores, voids, cast shrink, honeycomb damage

80 How to achieve valid POD data for RT - without destructive sectioning of natural flaws? Solution - Consensus evaluation of testset Inspectors differ on which are real flaws 24A/6R 21A/9R 16A/14R

81 Using baseline consensus to reduce variance Intersection of inspector agreement found valid in Metallurgical assessment 24A/6R 16A/14R 21A/9R

82 Example Five Variation Analysis Measuring known unknowns

83 All Solutions and POD analyses imperfect Control known knowns Measure known unknowns and account for in analysis

84 Example - EC inspection of aircraft lap splices -Per print fastners not all in a row Off axis model Parse new peaks based on offset Take the data and effort to understand what you can see varying There will be plenty which you can not!

85 Summary: Sources of variation in NDE processes Tight ellipse defines best practice for an NDE method against a defined flaw range Variation is the enemy, and is affected/controlled by: System Response Inspector decision variance Insp. to insp. variance Creeping procedural elegance Use of transfer function Master gaging Quant. cont. resp. model Operator fatigue Acquisition rate variance Interpolation/rounding variance Snowflake crack morphology Automated Calibration Soft rule based decision systems Automated data acquisition Flaw Size Ensure you account for these in your POD demonstration!

86 Summary & Closure Before the start: Background on NDE Reliability Measurement Practice Vetting and assessing existing databases of POD capability Completing experimental gap analysis Calibrations & transfer functions Protocols for partial/full capability demonstrations Quality checking the data Selecting a model Understanding capability relationships behind POD curves Understanding & addressing experimental variation Substantiating and presenting the results

87 Feedback: Did you achieve goals? Comfortable using typical POD software and presentation tools Can produce valid results Aware of sources of variation and mitigation steps

88 References/Resources ASM NDT handbook Mil Std 1823 ASQ Reliability Engineering Bible Minitab SPlus R Open source statistical software New release August 2009