Agitation sensor based on Facial Grimacing for improved sedation management in critical care

Agitation sensor based on Facial Grimacing for improved sedation management in critical care The 2 nd International Conference on Sensing Technology ICST 2007 C. E. Hann 1, P Becouze 1, J. G. Chase 1, G. M. Shaw 2 and X. Chen 1 1 Department of Mechanical Engineering, University of Canterbury, Christchurch, New Zealand 2 Department of Intensive Care Medicine, Christchurch Hospital, Christchurch, New Zealand 1

Introduction Agitation in Intensive Care Unit (disease, injury, life-support) Sedation Current methods to measure agitation are subjective (Hospital staff) Under sedation Reduce patient recovery Risk of injury (staff, self extubation ) from agitation Over sedation Longer weaning period Cardiovascular depression and death! Increased length of stay and cost Necessity of an accurate method to measure agitation 2

Previous work A physiological model of agitation and sedation pharmacodynamics has been developed [Rudge et al.] Significant improvements in simulation requires a sensor!!! Current agitation sensors : Blood pressure/heart rate variability [Chase, Starfinger et al.] Digital Imaging of whole body motion [Chase, Agogue et al.] Medium to large overall average movement Limited resolution (wide view, no detail) 3

A new approach towards Agitation Sensing Motivation: Higher resolution on face Detect subtle facial change early signs of agitation Integrate with current sensors more accuracy Facial expression used in Visual Analog Scale (VAS) Objectives: Develop software for measuring the degree of facial grimacing, which clinical staff (and the literature) have noted as a distinct sign of emerging patient agitation. 4

Experimental Setup Several simulations imitating a patient in Critical Care have been done Camera Video Area Patient Bed Patient (a) (b) A patient moving head with no expression A stationary patient with differing degrees of agitation A patient moving head and grimacing 5

Overview of proposed algorithm Video acquisition Detect head position Put boundary around face Segment the face Evaluate grimacing 6

Reference point tracking Detect head s position: Place artificial marker point - e.g. on ventilator Convert image to grayscale Restrict region of interest Smoothing Normalization Thresholding 7

Reference point tracking Clear unwanted object: (a) Step 1 Step 2 Step 1 Take image complement (b) Step 2 Makes all white area touching a border black (c) (d) 8

Reference point tracking Unwanted objects (Case (d)): First frame: Delete all objects > or < size of dot Following Frames: Keep the dot that is within a predefined tolerance from the dot in the previous frame 9

Reference point tracking Results: Frame 1 Dot trajectory Frame 280 Frame 530 10

Extracting contour of face Skin Recognition: Skin hue property Red/Green ratio Shown to be effective in literature - works on pale skin and dark skin equally effectively obtain features of face that represent grimacing with ventilator straight forward, without ventilator its harder! use skin recognition to find face 11

Extracting contour of face Contour extraction Simple fast method rather than e.g. snakes (computationally expensive) Initialization (requires check by user before proceeding) Contour defined with 50 points Plot for every point the Red/Green Ratio along the line A to B Distance A to B is defined A B A B Previously computed curve Tangent of the curve Segment analysed 12

Extracting contour of face Detect the point along the line A to B that best meets the criteria for the border of the face 1.8 1.7 Ratio after smoothing ratio before smoothing 1.6 Ratio Red / Green 1.5 1.4 1.3 1.2 Skin 1.1 No Skin 1 A Segment from point A to B B Patients stay on average 4 days in ICU Learning system could be used to improve the results 13

Extracting contour of face Results (1) (2) (3) (4) (5) (6) (7) (8) (9) No Grimacing Medium Grimacing High Grimacing 14 Position of the head Right Left Front

Facial grimacing Stationary Measure grimacing level: Segregate the face by two squares around the cheek Threshold the segregated area Count the number of extra pixels appearing due to grimacing No grimacing Frame Nº 6 Medium grimacing Frame Nº 340 Grimacing Frame N º 361 15

Facial grimacing Stationary Results Grimacing level as a function of the frame number after normalisation 1 0.9 Frame 150 GrimacingLevel 0.8 0.7 0.6 0.5 0.4 0.3 Frame 115 0.2 0.1 Frame 50 0 0 50 100 150 200 250 300 350 400 450 500 550 Frame Number 16

Facial grimacing Dynamic Face Segmentation: Fixed section below dot Frame 1 Frame 100 Frame 200 Above the eyebrow and below the mouth Frame 30 Frame 400 Frame 500 Using dot position Frame 600 Frame 700 Frame 800 and contour of face 17

Facial grimacing Dynamic Measure Grimacing Extra wrinkles High-Pass filter Edge detection Counting edge pixels Grimacing measure, which is normalized to calm state relative change (important as a lot of patients are elderly) 18

Facial grimacing Dynamic Angle correction calibration: Sequence of frame moving the head with a calm face is used 1350 1300 G = G a * x Edge detection Level 1250 1200 1150 1100 G: Grimacing measure G: Corrected grimacing measure a: Gradient of the line x: Distance between the initial position and the current position of the dot 1050 Extreme Left Position 0 Extrem Right Position Distance between initial position and current position of the white dot 19

Facial grimacing Dynamic Results: (1) (2) (3) (4) (5) (6) (7) (8) (9) 20

Facial grimacing Dynamic Results: 1.2 1 (2) (5) (6) (9) (3) Manually normalized to a known calm state (1 st 80 frames of record here) 0.8 Grimacing Level 0.6 0.4 0.2 0 (1) (4) (8) (7) -0.2 0 200 400 600 800 1000 1200 Frame Number 21

Future work Clinical validation of methods used Comparison of the computed agitation level based on grimacing with agitation graded by nursing staff using the Riker Sedation-Agitation Scale or similar clinically validated scale (e.g. VICS, Richmond, Glasgow COMA, ATICE, ) 22

Conclusions This research project has successfully investigated the image processing and software requirements for measuring the degree of grimacing and patient agitation in real-time. The goal is to develop methods for high resolution measurement of specific facial features that correlate to patient agitation and pain. Sensitive, Quantitative, Objective, and Accurate agitation assessment for enabling better sedation management. 23