ADVANCED TECHNIQUES FOR THE VERIFICATION AND VALIDATION OF PROGNOSTICS & HEALTH MANAGEMENT CAPABILITIES

Transcription

1 ADVANCED TECHNIQUES FOR THE VERIFICATION AND VALIDATION OF PROGNOSTICS & HEALTH MANAGEMENT CAPABILITIES James E. Dzakowic G. Scott Valentine Impact Technologies, LLC 00 Canal View Blvd Rochester, NY 463 Abstract: In recent years significant emphasis has been placed on developing prognostics and health management (PHM) technologies for military and industrial applications. These technologies include anomaly detection, diagnostics, and prognostics algorithms as well as higher level reasoning algorithms for identifying root cause failures and directing maintenance actions. The verification and validation of these PHM technologies is an important step in building and quantifying the confidence in the diagnostic and prognostic system. The cost to correct an error after a PHM system is fielded will dramatically increase, thereby effecting the need for verification and validation techniques. This paper introduces a methodology for verifying and validating the capabilities of detection, diagnostic and prognostic algorithms through an on-line metrics based evaluation. Key Words: Condition-based maintenance; forecasting; performance metrics; prognostics and health management; validation; verification; Introduction: In recent years, many Prognostics and Health Management (PHM) products have been released to market incorporating near real-time, automated monitoring, fault detection and isolation capabilities and advanced prognostic prediction capabilities. There is a growing need to develop standard methods for quantifying the performance and effectiveness of these tools. Verification and validation is an important step in quantifying performance capabilities and overall effectiveness in a PHM system implementation. It costs dramatically more to correct an error once a PHM system is put into operation therefore there is significant value in verifying and validating the system prior to deployment. In order to assess the accuracy of the PHM technologies, Impact Technologies is developing a PHM Verification and Validation (V&V) Test Bench. The Test Bench is software application that contains fault pattern operational data and ground truth health information for a library of components to be provided to PHM technologies from the sensors through the reasoners.

2 Figure : PHM Verification and Validation Layers Figure : Roles of Verification and Validation Definitions: Verification and validation are correlated entities with distinct definitions depending on the application objective and scope. Different industries use different definitions and approaches. We will use the following definitions. Verification: The process of determining that a PHM technology accurately represents the developer s conceptual description as a function of the design specifications. Validation: The process of determining the degree to which the PHM technology achieves the performance specifications within the system constraints and provides accurate results in the operational environment. Based on these definitions, a graphical representation of the process of verification and validation is illustrated in Figure. The figure identifies two types of systems: a Conceptual System and a Realized System. The Conceptual System is composed of the architecture, the algorithms, and the specifications of the system that describe a desired diagnostic and prognostic system as well as the conceptual representation of the physical system or process of interest. The Conceptual System is produced by analyzing and observing the physical components and the requirement inputs from the customer (such as the FMECA study). The Realized System is the operational PHM application and hardware that implements the Conceptual System. Figure shows that Verification deals with the relationship between the conceptual system and the realized system and that Validation deals with the relationship between the experimental measurements and reality. Verification, while not simple, is much less involved than the more complex statistical nature of validation. Generally software quality assurance tools can be used to address programming errors and verify the correctness of the software. Due to the availability of such verification tools, the remainder of this paper will focus on validation techniques. System Architecture: The PHM V&V Test Bench utilizes a standardized set of mathematical and business case metrics for evaluating the performance and effectiveness of PHM systems. The evaluation process is capable of assessing PHM technologies in terms of their ability to detect, diagnose and predict fault to failure progression of specific failure modes. Specific metrics for these capabilities include accuracy, reliability,

3 sensitivity, stability, risk, economic cost/benefit, and robustness. The evaluation methodology shown in Figure 3 utilizes the set of metrics in conjunction with a database of testing, simulation or in-service fault data from different subsystems. The technical performance and accuracy of the PHM algorithms at the component or subsystem level are evaluated with performance metrics, while system level capabilities in terms of achieving the overall operational goals and economic cost/benefit are evaluated with effectiveness measures. Prognostics Health Assessment Decision Support Faulty Uncertain Healthy Condition Monitor Internet or TCP/IP Network Presentation Accelerometer Data Band-Pass Filter Band-Pass Filtered Signal Half-wave or Full-wave Rectifier Envelope Detector Rectified Signal Peak-Hold Smoothing Envelope Detected Signal Data Manipulation N 4 N ( ri r ) i= NA4 = m N ( rij rj ) m j= i= Data Acquisition Traditional Sensor Smart Sensor Figure 3: PHM V&V Test Bench Architecture Figure 4 OSA-CBM Architecture Enabling Technologies: The PHM V&V Test Bench relies on open systems architecture standards to enable communication between the Test Bench and the candidate PHM systems. Elements of this interface include monitoring system information (time domain data and configuration information) as well as diagnostic and prognostic information (anomaly status, diagnosis, fault severity, and confidence level). At the present time, standards for the exchange of these types of information are developing. Open systems architecture (OSA) standards offer the means to facilitate the development and integration of modular information analysis tools. The standard known as OSA for Condition Based Maintenance (OSA-CBM) consists of an abstract (technologyindependent) interface specification that is easily translated into XML implementations. The PHM V&V Test Bench has been designed with the OSA-CBM interfaces to allow developers to utilize know communication interface methods. In general, the OSA-CBM software architecture is broken down into many functional layers from sensing and data acquisition all the way up to the decision support and presentation layers, as shown in Figure 4. This architecture models all the sensor, processing and presentation layers necessary for a comprehensive and integrated PHM system. OSA-CBM does not impose any requirements on the internal structure of compliant software modules. The architectural constraints are applied to the structure of the interfaces and to the behavior of the modules. This approach allows complete

4 protection of proprietary algorithms and software design approaches within the software module. The PHM V&V Test Bench architecture relies on three specific interface layers of the OSA-CBM model: Data Acquisition, Health Assessment, and Prognostics Assessment. The Data Acquisition (DA) layer function will be performed by the test bench. In response to a PHM request, the DA output is passed to the PHM algorithm as an XML file. When completed, the PHM algorithm will return Health Assessment (HA) and Prognostics Assessment (PA) information to the test bench in the XML format. Figure 5 shows how the Test Bench software and the vendor s PHM module fit into the OSA- CBM architecture. Figure 5: PHM Module Interface Data Identification: The data sets and PHM metrics utilized within the V&V simulation environment include capability to assess fault detection, isolation and prediction capabilities. There are several methods for obtaining data for use in PHM validation. Table identifies several methods of data collection and describes benefits and drawbacks of each. This data is passed from the PHM V&V Test Bench database to a candidate PHM algorithm via the aforementioned DA XML file.

5 Table : Data Sources Method Description Benefit Drawback Simulation Models and Fault Generation Component and LRU Fault and Failure Tests Subsystem/System Fault Characterization Tests Limited Vehicle Data Evaluation Technology Maturation Field Program Signals, noise and fault signatures simulated mathematically Use data collected on component or subscale test rigs. Tests on stands using actual subsystem hardware Dedicated missions for normal and off-normal test and evaluation Data collection and tests during vehicle service Ideal for generalizing (varying) signals and noise; partially overcomes small sample problem Real seeded fault, natural fault, and fault progression data available at little cost to program Data may not always be realistic; model may be expensive to develop to reliably simulate. Limited applicability for actual system fault observability. Somewhat realistic data; Seeded faults may not be usually well controlled and entirely realistic of natural documented with ground truth fault. information. Realistic data of normal and some performance faults. Covers full range of vehicle mission, operational, and environmental conditions Expensive; doesn t cover full range of potential conditions; no critical faults or progression. Limited to opportunistic fault occurrence. Delays use and implementation in field Ground Truth Identification: Ground truth is representative of the actual condition of the system and serves as a baseline to compare the results from PHM algorithms. The validation of PHM systems at all levels (sensors, algorithms, fusion and reasoner) relies on ground truth information. Ground truth is typically an engineering estimation of the actual condition, rather than an absolute truth as the name implies. It may be based on visible evidence, a separate feature or prediction result in the data, or a damage severity curve. The appropriate selection of ground truth is vital in the validation process. Having a warehouse of data sets with associated ground truth enables PHM decision makers and developers to obtain validation data sets and evaluate the performance of proposed algorithms. Metrics Based Evaluation: Figure 6 lists important performance and effectiveness metrics that are being developed for fault detection, diagnostics and prognostics. These PHM capability metrics will capture system implementation and cost issues. For all of the developed metrics, a low score indicates an undesirable result. For example, a high computational resource requirement score is awarded to PHM algorithms that use a small portion of the computer s resources. A few of the specific metrics for detecting and classifying system faults are discussed followed by prognostic metrics.

6 Figure 6: Technology Performance Metrics Diagnostic Health Assessment Metrics: An assessment of the detection confidence level over the entire severity range is achieved through the Accuracy metric. A Detection Threshold Metric measures a PHM algorithm s ability to identify anomalous operation associated with incipient faults with a specified confidence level. Confidence levels of 67% and 95%, corresponding to one and two standard deviations, are used to calculate the detection threshold metric. Accuracy = C( s) ds 0 Detection Threshold = S( c) where: C(s) = the success function and S(c) = ground truth severity corresponding to success function value c () () The success function of the diagnostic tool is a relationship between the average confidence and the average severity level. Note that this relationship may be used to assess either Boolean (0 or ) confidence levels or continuous confidence levels within the same interval. A PHM technology whose output fault confidence level fluctuates wildly is difficult to interpret and is therefore undesirable. For example, a diagnostic tool that produces a Boolean result of either no fault or fault may flicker as the fault severity approaches the detection level. A Stability Metric measures the range of confidence values that occur over the fault transition by integrating the peak to peak difference at each point in the transition. In addition, PHM systems should detect anomalies over the full range of operating (duty) conditions such as loads, speeds, etc. A Detection Duty Sensitivity Metric measures the difference between the outputs of a PHM algorithm under various duty conditions. Stability = ( C ( s) C ( s) )ds (3) 0 H L

7 X% X% Accuracy Accuracy X% Accuracy Duty Sensitivity = ( C ( s) C ( s ) ds 0 ) (4) where: C H (s) = maximum value of the success function at severity s C L (s) = minimum value of the success function at severity s and C (s) = success function at duty condition C (s) = success function at duty condition In an operational environment, sensor data is sometimes contaminated with noise that may interfere with the operation of diagnostic algorithms. The robustness of a PHM algorithm to noisy data will be measured by a Noise Sensitivity Metric. NoiseSensi tivity = - C s C s ds ( FalsePosit ive) ( ( ) ( )) * (5) 0 where: C (s) = success function under noise condition C (s) = success function under noise condition s = severity Prognostic Assessment Metrics: There is currently no general agreement as to a standard and acceptable set of metrics that can be effectively employed to assess the technical performance of prognostic systems. Some initial developments have arisen in this area with a class of metrics that stem primarily from the main objectives of prognosis and serve to assess the uncertain (statistical) nature of prognostic algorithms. With such metrics, the concepts associated with prediction of performance degradation in components must be addressed along side metrics for hard failures associated with a component. The uncertainty concept for hard failures is shown in Figure 7. Hazard Lines % P f Hazard Lines Efficiency X% Precision 86% Present MTTF Prediction X% Confid. Figure 7: Failure Prognostics and Uncertainty Concepts used in Development of Metrics Time 85% Present Degradation Prediction X% Confidence Time Figure 8: Performance Degradation Prognostics and Uncertainty Concepts The uncertainty concept for performance degradation prognostics is shown in Figure 8. As can be seen from these figures, a similar set of consistent metrics should be able to be developed that can be applied to both based on the concepts described next.

8 First, in order to set the stage for our prognostic metrics, we will define the mean and standard deviations for the prediction on time to failure or unacceptable degradation level as follows. N Mean: E{ t pf } = t pf () i N i= N Standard Deviation: { t } ( t () i E{ t }) S (7) df pf pf N i= = (6) Figure 9 and Figure 0 illustrate the proposed method for determining accuracy of a prediction. It is a measure of how close a point estimate of failure time is to the actual failure time, and that is defined by Accuracy = DC ) R ( y real e (8) Figure 9: Determining Accuracy of Prediction Figure 0: Determining Uncertainty of Prediction Precision is a measure of how narrow of an interval the remaining useful life predictions fall. Confidence is the probability of the actual remaining life falling between the bounds defined by the precision. Figure depicts the determination of precision and confidence bounds. Assume that a prognostic algorithm predicts the following progression or evolution of a fault with the associated uncertainty bounds: At current time, t o, the prognostic routine is predicting a mean time-to-failure of T fm, an earliest time to failure of T fe, and a latest time to failure of T fl, as shown in Figure. The hazard line (with associated uncertainty) specifies the fault magnitude (dimension) at which the component ceases to be operational (failure).

9 Figure : Determining Precision and Confidence Figure : Confidence Limits and Uncertainty Let us assume that failure data is available upon which to superimpose a distribution or distributions, as shown pictorially in Figure 3. Strictly speaking, these distributions are either possibility (fuzzy) functions or probability density functions. Suppose that the distribution at T fm (when fault magnitude crosses the hazard line) is shown in Figure 4. Figure 3: Possibility Density Functions for Confidence Figure 4: Distribution at the Hazard Line For this case, assume that the pre-planned mission is estimated to require the availability of the asset under consideration until time T. The integral under the distribution curve from T to infinity will give us an estimate of the confidence level in terms of probability that the asset will not fail before the mission is completed. Consider next, the same distribution. Now we specify a certain confidence level, say 95%, and would like to find the time T when we are 95% confident that the component will not fail. Integrating the distribution curve T to infinity and setting the result equal to the confidence limit (95% in our example), we solve for T thus arriving at the length of time (starting at T o ) that asset will be available given the desired confidence limit. We view this procedure as a dynamic evolution of our estimates. That is, as more data becomes available and time marches on, new confidence limits are derived and the uncertainty bounds shrink through appropriate learning routines. The procedure outlined above may lead, eventually, to specification of performance metrics for prognostic

10 systems. Such additional metrics may refer to the reliability of the confidence limits, risk assessment, safety specs, and others. Cost Assessment: Acquisition and implementation costs of the PHM algorithm under investigation may have a significant effect on the overall system s cost effectiveness. An Implementation Cost Metric is proposed that will simply measure the cost of acquiring and implementing a PHM system on a single application. If the PHM system is applied to several pieces of equipment, any shared costs are divided among them. Operation and maintenance costs of the PHM algorithm may also play a significant role in determining whether a PHM system is cost effective. An O&M Cost Metric will also be needed to measure the annual cost incurred to keep the PHM system running. These costs may include manual data collection, inspections, laboratory testing, data archival, re-licensing fees and repairs. The ability of the PHM system to be run within a specified time requirement and on traditional aerospace computer platforms with common operating systems is important when considering implementation. Therefore, a metric that takes into account computational effort as well as static and dynamic memory allocation requirements is necessary. A Computer Resource Metric will be developed to compute a score based on the normalized addition of CPU time to run (in terms of floating point operations), static and dynamic memory requirements for RAM and static source code space, and static and dynamic hard disk storage requirements. Complex systems are generally more susceptible to unexpected behavior due to unforeseen events. Therefore, a System Complexity Metric will be used to measure the complexity of the PHM system under investigation in terms of the number of Source Lines Of Code (SLOC) and the number of inputs required. The benefits achieved through accurate detection, fault isolation and prediction of critical failure modes will be weighed against the costs associated with false alarms, inaccurate diagnoses/prognoses, and resource requirements of implementing and operating specific techniques. The simplified cost function shown below states the Technical Value provided by a specific diagnostic or prognostic technology for a particular system. The value of a PHM technology in a particular application is the summation of the benefits it provides over all the failure modes that it can diagnose or prognose less the implementation cost, operation and maintenance cost, and consequential cost of incorrect assessments as stated in Total Value equation layout as savings-costs. Technical Value P ( D * α + I * β ) ( P )*( P * φ P * θ ) where: = (9) f * f D I P f = Probability (time-based) of occurrence for a failure mode D = Overall Detection Confidence metric score α = Savings realized by detecting a fault prior to failure I = Overall isolation confidence metric score β = Savings realized through automated isolation of a fault P D = False positive detection metric score

11 φ = Cost associated with a false positive detection P I = False positive isolation metric score θ = Cost associated with a false positive isolation Total Value = Technical Value A O ( where: Failure Modes i P c )* δ (0) A = Acquisition and Implementation Cost O = Life Cycle Operation and Maintenance Cost P c = Computer Resource Requirement score δ = Cost of a standard computer system Summary: This paper provided an overview of the methodology incorporated in a PHM Verification and Validation application that is being developed by Impact Technologies in cooperation with Georgia Institute of Technologies. The application contains diagnostic and prognostic V&V metrics, a central location for access to costly seeded fault data sets and example implementations, and an automated tool for impartially evaluating the performance and effectiveness of PHM technologies. Analysis of the effectiveness of PHM technologies at each level provides the verification and validation to measure performance as well as a valuable feedback mechanism for ongoing development of PHM systems. Prognostics and Health Management is still a relatively new field, with advancements in technologies continually being made. The need for metrics to measure and validate performance will provide benefit to many current military and commercial acquisition programs. References:. Roemer, M., Dzakowic, J., Orsagh, R., Byington, C., Vachtsevanos, G., Validation and Verification of Prognostic and Health Management Technologies, IEEE Aerospace, Byington, C., Roemer, M., Kalgren, P., Vachtsevanos, G., Verification and Validation of Diagnostic/Prognostic Algorithms, MFPT Aguilar, R., Chuong, L., Santi, L., Sowers, T., Real-Time Simulation for Verification and Validation of Diagnostic and Prognostic Algorithms, AIAA Joint Propulsion Conference, July Bodden, D., Hadden, W., Grube, B., Clements, N., PHM as a Design Variable in Air Vehicle Conceptual Design, IEEE, Overman, R., Collard, R., The Complimentary Roles of Reliability-Centered Maintenance and Condition Monitoring, IMC, OSA-CBM Data Modules (version.0) XML Schemas,