Feedback control of the forearm movement of tetraplegic patient based on Microsoft Kinect and multi-pad electrodes

Transcription

1 JOURNAL OF AUTOMATIC CONTROL, UNIVERSITY OF BELGRADE, VOL. 21:7-11, 2013 Feedback control of the forearm movement of tetraplegic patient based on Microsoft Kinect and multi-pad electrodes Matija D. Štrbac, Nebojša M. Malešević, Radoje Čobeljić and Laszlo Schwirtlich Abstract We present a novel system for control of elbow movements by electrical stimulation of the biceps and triceps in tetraplegic patients. The operation of the system uses the novel algorithm and applies closed loop control. Movement of the arm is generated via multi-pad electrodes developed by Tecnalia Serbia, Ltd. by the stimulator that allows asynchronous activation of individual pads. The electrodes are positioned over the innervation of the biceps and triceps muscles on the upper arm. This layout allows distributed activation; thereby, selective and low fatiguing activation of paralyzed muscles. The sensory feedback comes from the image acquired by the Microsoft Kinect system and the depth stream analysis is performed in real time by the computer running in the MatLab environment. The image based feedback allows control of the hand position at the target by cocontraction of the antagonists. The control adjusts the stimulation intensity and results with the tracking of the desired movement. The algorithm was proven to operate efficiently in a tetraplegic patient. I. INTRODUCTION FUNCTIONAL ELECTRICAL THERAPY (FET) integrates intensive exercise, functional electrical stimulation and motivation to relearn affected motor functions, e.g. manipulation and grasping that are absent as a result of the stroke. Results presented in the literature suggest the FET has favorable carry over effects [1-2]. Results also suggest that it would be beneficial to add automatic selection of the stimulation pattern for the task intensive exercise. The camera based artificial visual perception system which automatically selects the pattern of stimulation that allows automatic recognition and classification of objects used during the FET has been recently tested. Several studies [3-5] proved that different systems and methods could be used to solve the task of object identification and automatic grasp type selection. Although these systems proved almost faultless in grasp type classification process, due to the limitations of sensors used in [3-5], they featured low Manuscript received August 31st, This research was partly supported by the Ministry of Education and Science, and Technological Development, Republic of Serbia (Project no ) and Tecnalia Research and Innovation, San Sebastian, Spain. M. D.Štrbac (corresponding author) is with Tecnalia, Serbia and Faculty of Electrical Engineering, University of Belgrade, Serbia. (Phone: , Fax: , matija@etf.rs). N. M. Malešević is with Tecnalia, Serbia and Faculty of Electrical Engineering, University of Belgrade, Serbia. R. Čobeljić is with Rehabilitation Clinic 'Dr Miroslav Zotović', Belgrade, Serbia. L. Schwirtlich is with Rehabilitation Clinic 'Dr Miroslav Zotović', Belgrade, Serbia and the State University of Novi Pazar, Serbia. DOI: /JAC S precision and low robustness in hand position estimation and therefore could not meet several clinical application requirements. The Microsoft Kinect is an effective range camera which tracks movement in (3D) space. One of the advantages of the Microsoft Kinect is that it allows the estimation of the distance and pattern of the moving objects in the scene compared to standard cameras. Microsoft Kinect found many applications (e.g., robotics and virtual reality [6], education and healthcare, and physical rehabilitation [7-10]). Kinect was already used as a sensory input, along with the electrogoniometers, in the GO-SAIL electrical stimulation system and according to preliminary results this system successfully applied stimulation to three muscle groups to include the wrist and hand to supplement activity and promote the successful completion of a range of functional task [11]. The applicability of Microsoft Kinect as the only sensory input for the closed loop control of the FES system was still to be validated. Our previous study [12] showed that this device can be included in the system setup commonly used in FET to track the hand 3D coordinates in real time with position estimation error that is less than 1cm. These estimates could be used for automatic control of the stimulation parameters in the reaching and/or the grasping phase. Studies have shown that paretic subjects with innervated elbow extensor muscles can benefit from grasping devices only if, in addition, their reaching movements can be restored [13], and therefore the rehabilitation of the reaching movements is in some cases even more important than the grasping phase, alone. Methods based on mathematical model of the human arm [14] and further computational optimization of the control algorithm [15] proved to be too complicated and subject specific to be used in practice and heuristically methods were more often used to define control strategy. In past many different upper limb control algorithms that incorporate biological synergies in order to guide reaching in FET were proposed [16-17]. Approach that consists out of accelerometer for gathering sensory data and artificial neural network decision making system for the control algorithm was also tested [18]. In this paper we present results of closed loop control of forearm movement in transverse plane based on artificial perception. Kinect serves as the only sensory input of our system. Image processing and computer vision algorithms estimate the hand position, and based on that information, the current intensities and stimulation patterns are calculated. From the algorithm output, commands are wirelessly sent to multi-pad FES system built by Tecnalia Serbia Ltd. [19] which activates patient s biceps and triceps accordingly. This feedback control program was run on personal laptop (Intel

2 8 ŠTRBAC M, MALEŠEVIĆ N, ČOBELJIĆ R, SCHWIRTLICH L, FEEDBACK CONTROL OF THE FOREARM MOVEMENT... Core i5 2.4 GHz, 4 GB of RAM) and it worked in real time with 12 frames per second refresh rate. Our system was initially tested on 5 healthy volunteers and after promising results the methodology was validated on a tetraplegic patient with paralyzed; yet innervated biceps and triceps muscles (C5, incomplete spinal cord lesion, ASIA score A). The selection of the patient was based with the assumption that the application of electrical stimulation in subjects who have voluntary contractions could be misleading. Namely, it would be impossible do discriminate between the natural control and the contribution of electrical stimulation. A. Image processing Figure 1. Kinect in the defined system setup II. METHODS The Microsoft Kinect sensor consists of an infrared laser emitter, an infrared camera and an RGB camera and as an output forms two image streams, one of them consisting of the RGB images while the other is depth matrix with the 11 bit amplitude resolution. To ensure easy inclusion of the Kinect sensor in the FET environment and to optimize the working area in our experiment, the sensor is placed on the 0.6 meter high camera stand on top of the working table, overlooking the table with around 60 degrees tilt in respect to the transverse plane in our system setup. Kinect is set to work in near mode (which provides a range of 500mm to 3000mm of the measured depth image data) and all the image processing algorithms are done over the 320x240 pixels depth stream images. After initial calibration [20] of the sensor, real world coordinates of every pixel in the depth image matrix are calculated. Algorithm for hand tracking is based on defining the working area, i.e. the table by which the patient is seated. The geometry of the table dictates that real world coordinates of all the pixels that belong to the table must meet the plane equation, and therefore, the table pixels can be easily distinguished with Random Sample Consensus (RANSAC) [21] algorithm. Basic RANSAC would consist out of forming a random distribution of N triplets of points across the image pixels that would define N planes, and calculating number of image pixels that belong to every of the N planes. Thanks to the fact that the table takes more than 50% of the image, this algorithm step was optimized by clustering plane vectors and finding the central plane in the largest cluster based on the angle in between them, as described in our previous study [12]. After the program starts the first 10 image frames from the depth stream of the Kinect sensor are used to form the background image, i.e. image of table pixels. Bearing in mind that it would be very difficult for tetraplegic patient to keep his hand away from the table after every trial and move it in after the background is formed, the use of previously formed background images is also enabled. In our previous study [12] an algorithm that enables real time hand tracking and object detection was presented. The condition that we introduce is that there are no moving objects in the scene except the arm/hand. If this assumption is satisfied the hand primitive can be extracted by subtraction of the RANSAC results when the hand is moving above the table and the formed or selected background image. For reference point on the extracted hand the median pixel in the heuristically defined area around the tip of the hand is used. Coordinates of this point are used for geometrical transformation of coordinate system to the origin point and also as sensory input for the control algorithm in later calculations. Hand velocity is calculated as the difference between the current hand position and the one in the previous image divided by the elapsed processing time, and is also used in control algorithm. When the subject is ready, the coordinate system is calibrated to the hand reference point and the stimulation is started to run concurrently with the image processing and hand tracking. B. Stimulation procedure For stimulation we used INTFES (INTelligent Functional Electrical Stimulation TECNALIA) multi-pad electrode system [17]. Main feature of this system is ability to individually activate specific pads within multi-pad electrode and independently set current amplitude for each active pad, defining stimulation pattern. It can dynamically change stimulation patterns based on control algorithm. Other important feature of system based on multi-pad electrodes is possibility of employing fatigue resistant stimulation protocol. Using distributed low-frequency stimulation, it is posible to prolong periods of functional stimulation which is important for FET [22]. For the stimulation of upper extremities surface-distributed lowfrequency asynchronous stimulation can double the time interval before the onset of fatigue compared with conventional synchronous stimulation [23]. Our stimulation protocol takes into account recomended pulse distibution paradigm and deliverstime delayed pulses to selected pads. Multi-pad electrode design is appropriate for specific function and electrode placement. For present protocol we utilized 2 electrodes in 4x4 configuration (Fig. 2), positioned over biceps and triceps muscles. 2 anodes were positioned near patient's elbow. Optimization of active electrode pad configuration, based on visual feedback, for elbow flexion and extension was deternmined at the beginning of protocol, and preferred patterns are forwarded for feedback control algorithm. PC with Matlab application serves as Host controller which emulates stimulation paterrns in real-time via Bluetooth link with stimulator. Current amplitudes range in this protocol was 0-20 ma, pulse width 250 μs, pulse train frequency 40 Hz and interpulse delay in n-lets 1 ms.

3 JOURNAL OF AUTOMATIC CONTROL, UNIVERSITY OF BELGRADE, VOL. 21, Figure 2. INTFES stimulation system: Multi-pad electrode and stimulator C. Feedback control algorithm The goal of the algorithm for automatic control of forearm movement is to find current intensities needed for stimulating of elbow extension, flexion and to maintain preferred position of patient's hand using closed loop control based on real-time visual feedback. Knowing that only one degree of freedom can be controlled with biceps and triceps stimulation, we decided to regulate the position down the x axis. Main reason for this is because y axis position can be easily regulated by inclining the body forward and backward. In initial position, patient's hand is slightly bent as it is most comfortable position which most of patients can maintain. The control algorithm is divided in three stages. At the beginning, all current amplitudes are set to 0 ma. In first stage, algorithm is incrementing currents on pads stimulating triceps. This process is done in steps of 1 ma every 300 ms, until consistent movement of forearm with respect the upper arm is detected. This movement is defined as movement with hand velocity above 15 cm/s threshold. Maximal current is saved as triceps current (TC). Second stage of algorithm starts when elbow reaches full extension. Then, amplitudes on set of pads for triceps stimulation are set to 0 ma, and incrementing of current amplitudes for biceps stimulation starts. Incrementing of stimulation current amplitude is stopped upon detection of forearm steady movement, i.e. the hand velocity is above the threshold. After reaching initial position algorithm enters third stage. Goal of this stage is to steer hand towards position defined as target. In our protocol, target position is calculated by range of motion divided by 2. Third stage (closed loop control) begins with setting the triceps stimulation current to the value saved in first stage. Basic mechanism for control is based on hand position in respect to target position. While in range: 0 cm (initial position) to target position - 1 cm, triceps stimulation is incremented while biceps currents are decremented. In range between target - 1 cm and target + 1 cm no changes in current amplitudes are made. When hand position is over target + 1 cm, triceps currents are decremented while biceps currents are incremented. Figure 3. Hand trajectory during stimulation protocol. Red line represent first algorithm stage, blue second, and green third v < v t ; TC + + Stage 1 = v > v t ; TC save p > p e ; Stage 2, TC = 0 v > v t ; BC + + Stage 2 = v < v t ; BC save p < 0; Stage 3, TC p < p t 1 cm; TC + +, BC Stage 3 = p t 1 p p t + 1; nothing p > p t + 1; BC + +, TC Function for elbow control, where v is velocity of patient's hand, v t velocity threshold, TC stimulation current amplitudes applied on patient's triceps, p hand position, p e position of full elbow extension, p t position of target and BC stimulation current amplitudes applied on patient's biceps. III. RESULTS A. Healthy subjects Several trials on 5 healthy subjects (age 22-26) proved that our algorithm was able to identify adequate biceps and triceps stimulation current to produce the movement in the desired direction. The automatic regulation of the current amplitudes in the closed loop control stage was good enough to maintain the hand in the target position by cocontraction, even in the cases when there were disturbances due to fatigue or voluntary contractions. The range of movement and the destination point, i.e. a, b and c, were selected individually for every subject. Target position was set to one half of the range of motion. On Fig. 4 several trials for 2 subjects are presented. It can be noticed that some trajectories include one or two additional overshoots and this is due to the nonlinearity and

4 10 ŠTRBAC M, MALEŠEVIĆ N, ČOBELJIĆ R, SCHWIRTLICH L, FEEDBACK CONTROL OF THE FOREARM MOVEMENT Subject 1 trials Hand reference point [cm] Target time [s] 40 Subject 2 trials Hand reference point [cm] Target 0 Figure 4: Hand trajectory for several trials on two healthy subjects time [s] delay of the muscle activation and inability of our closed loop control algorithm to respond correctly. In spite of this, it is clear that all the trajectories gravitate towards target position. The time needed to reach this position is in between 5 and 10 seconds for all trials. In that process the biceps and triceps current amplitude identification lasts for around 5 seconds, and in the third stage (closed loop control) 2 to 4 seconds is needed to guide the hand to the point of interest. Trajectories for the other 3 subjects are analogous to the presented results and are for simplicity and clarity not presented in the paper. Subjects were never informed about the target position and were not presented with the feedback about the hand position during the experiment, but due to existence of voluntary muscle activation in the overall muscle activity, these results were not sufficient to claim that presented system is able to control the hand position on its own. Healthy subject could influence the hand trajectory with the voluntary muscle activity and by this means improve or spoil the measurements. This is the reason why these results had to be validated on a tetraplegic patient in the absence of voluntary movements. B. Tetraplegic patient This system and feedback control algorithm was validated on one tetraplegic patient with an incomplete C5 lesion in rehabilitation clinic "Dr Miroslav Zotović", Belgrade. The subject agreed to the informed consent which was approved by the local ethics committee. Experiment consisted of several trials with a different starting point for the stimulation process. Again, the range of movement and the destination point were selected in respect to the patient s anatomy. The range of motion was 15 cm and the target point was set to 7 cm, in x axis direction. Again, for simplicity, results of only one trial are presented in this study. Stimulation patterns and resulting movement are fairly similar to the presented for all the other recorded datasets. Figure 5: One trial of feedback control of the tetraplegic patient forearm movement in transverse plane with presented algoritm On Fig. 5 we present the regulated coordinate in the estimated hand position and the corresponding stimulation currents on the biceps and the triceps electrodes during one trial. The amplitude needed to move the triceps with desired velocity was 13 ma, while for biceps movement only 6 ma current was used. After defining these currents the third stage (feedback control) starts with these values and they are changed in a predefined small range in order to put the hand in the target position and keep it there by cocontraction. In presented trial it took less than two seconds to achieve this with only one overshoot caused by higher amplitude of the triceps stimulation. The hand was maintained in the target position for more than 15 s with cocontraction, and after that presumably because of the muscle fatigue the higher stimulation of the triceps and lower of the biceps is required. During the last 10 s of the stimulation our algorithm responded on this coordinate changes twice with the proper stimulation amplitude modifications and the hand was brought back and kept it in the target position. IV. CONCLUSION In this paper we presented close loop system that can be used for arm control of tetraplegic or other patients with impaired elbow control. Use of camera system and stimulation platform capable for dynamic change of shape, position and size of stimulation electrode together with stimulation parameters/patterns is envisioned as advanced therapeutic device for clinical and home use. Aim of this pilot study, consisted out of healthy subject tests and validation on a tetraplegic patient, was to evaluate effectiveness of control algorithm used to produce desired hand movement. We emulated stimulation parameters in real-time to minimize perturbations which are inevitable over time. The results presented show that the new control performs the task efficiently: setting the stimulation intensity of a pair of antagonist muscles that results with: control of elbow flexion/extension, reaching a target position by hand and postural assistance to the arm that keeps the hand at the target position.

5 JOURNAL OF AUTOMATIC CONTROL, UNIVERSITY OF BELGRADE, VOL. 21, It is important to notice that the first two stages in our algorithm serve for identification of for biceps and triceps stimulation amplitudes that will result with adequate hand movement. After these amplitudes are identified, the first two stages of the algorithm can be omitted in every subsequent task. Closed loop control algorithm will be able to guide the hand towards any new target position in the range of motion of the subject, and keep it there following the same rules. In this manuscript, we wanted to present the results of our method in entirety, thus presenting that our algorithm is able to establish the stimulation current amplitudes that will produce hand movement and result with the positioning of the hand at the point of interest. In future work, we will also investigate advanced closed loop control algorithms that are based on mathematical model of the human arm in order to produce more natural-like hand movement profile. ACKNOWLEDGMENT The research was conducted at the Laboratory for Biomedical Engineering and Technology ( at the Faculty of Electrical Engineering, University of Belgrade and Rehabilitation Clinic "Dr Miroslav Zotović", Belgrade. The authors express their appreciation for suggestions and guidance to Prof. Dejan B. Popović. We appreciate and acknowledge the assistance by Prof. Ljubica Konstatinović, M.D., School of Medicine, University of Belgrade and Prof. Mirjana B. Popović from the Faculty of Electrical Engineering, University of Belgrade in the clinical works. REFERENCES [1] M. B. Popović, D. B. Popović, T. Sinkjær, A. Stefanović and L. Schwirtlich, Restitution of reaching and grasping promoted by functional electrical therapy, Artificial Organs, 26(3): , [2] M. B. Popović, D. B. Popović, T. Sinkjær, A. Stefanović and L. Schwirtlich, Clinical Evaluation of Functional Electrical Therapy in Acute Hemiplegic Subjects, J Rehab Res Develop, 40(5): , [3] Đ. Klisić, M. Kostić, S. Došen and D. B. Popović, Control of Prehension for the Transradial Prosthesis: Natural-like Image Recognition System, Journal of Automatic Control, 19: 27-31, [4] S. Došen and D. B. Popović, Transradial Prosthesis: Artificial Vision for Control of Prehension, Artificial Organs, 35(1):37-48, [5] M. Štrbac and M. Marković, Stereovision system for estimation of the grasp type for electrotherapy, Serbian Journal of Electrical Engineering, 8(1):17-25, [6] Y.-J. Chang, S.-F. Chen and J.-D. Huang, A Kinect-based system for physical rehabilitation: A pilot study for young adults with motor disabilities, Research in Developmental Disabilities, 32(6): , [7] H.-m. J. Hsu, The Potential of Kinect in Education, International Journal of Information and Education Technology, 1(5): , [8] A. P. Lanari Bo, M. Hayashibe and P. Poignet, Joint Angle Estimation in Rehabilitation with Inertial Sensors and its Integration with Kinect, In Proc Annual Conf Engineering in Medicine and Biology Society (EMBC), Aug Sept , pp , doi: /IEMBS, [9] E. Hallewell, T. Exell, K. Meadmore, C. Freeman, M. Kutlu, A.M. Hughes, and J. Burridge, Goal-orientated Functional Rehabilitation using Electrical Stimulation and Iterative Learning Control for Motor Recovery in the Upper Extremity Post-Stroke, In Proc 18 th IFESS Annual Conference 2013 pp. 37. Academic Mind, ISBN [10] M. Štrbac, M. Marković, and D. B. Popović. "Kinect in neurorehabilitation: computer vision system for real time hand and object detection and distance estimation." in Proc 11 th Neural Network Applications in Electrical Engineering (NEUREL), Sept. 2012, pp: , doi: /NEUREL [11] D.B. Popović, and M.B. Popović. "Nonanalytical control for assisting reaching in humans with disabilities." in and Jack M. Winters. (Ed.) Biomechanics and Neural Control of Posture and Movement, pp , Springer Verlag, New York, 2000 [12] M. Popović, N. Jovanović, and L. Schwirtlich. "Identification of parameters for control of assistive systems." J. Aut. Control 4(1) [13] K. M. Jagodnik, and A. J. van Den Bogert. Optimization and evaluation of a proportional derivative controller for planar arm movement. J Biomechanics, 43(6), , [14] Popović MB, Popović DB. A new approach to reaching control for tetraplegic subjects, J Electromyog Kinesiol, 4: , [15] M. B. Popović, and D. B. Popović. Cloning Biological Synergies Improved Control of Elbow Neuroprostheses. IEEE EMB Magazine, 20(1):74-81, [16] B. Mijović, M. B. Popović, and D. B. Popović. Synergistic control of forearm based on accelerometer data and artificial neural networks. Brazilian J Med Biol Res, 41(5): , [17] N. M. Malešević, L. Z. Popović Maneski, V. Ilić, N. Jorgovanović, G. Bijelić, T. Keller, and D. B. Popović. A Multi-Pad Electrode based Functional Electrical Stimulation System for Restoration of Grasp J Neuro Eng Rehab, 9:66, [18] K. Khoshelham and S. O. Elberink, Accuracy and Resolution of Kinect Depth Data for Indoor Mapping Applications, Sensors, 12(2): , [19] M. Fischler, and R. C. Bolles. "Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography." Communications of the ACM, 24(6): , [20] N. Malešević, L. Z. Popović, L. Schwirtlich, and D. B. Popović. Distributed low-frequency electrical stimulation delays muscle fatigue compared to conventional stimulation. Muscle & Nerve, 2010, 42: , DOI /mus [21] L. Z. Popović-Maneski, N. Malešević, A. Savić, D. B. Popović, Spatially distributed asynchronous stimulation delays muscle fatigue. Muscle & Nerve, /mus.23840, 2013.