ANALYSIS OF MOTION-BASED CODING FOR SIGN LANGUAGE VIDEO COMMUNICATION APPLICATIONS

Transcription

1 ANALYSIS OF MOTION-BASED CODING FOR SIGN LANGUAGE VIDEO COMMUNICATION APPLICATIONS Benjaporn Saksiri*, Supavadee Aramvith**, Chai Phongphanphanee* *Information System Engineering Laboratory, Department of Computer Engineering, Chulalongkorn University, Bangkok 10330, Thailand Tel: (662) , Fax: (662) **Department of Electrical Engineering Chulalongkorn University, Bangkok 10330, Thailand Tel: (662) , Fax: (662) Abstract Digital video communications have been characterized by an exponential growth in the last decades. The continuous growth causes an evolution of several new applications such as video conferencing, video , mobile video, video streaming over internet, and sign language video communication, etc. To transmit video over internet or wireless networks, such applications needs digital video coding algorithm such that video could be compressed to meet bandwidth requirements. In this paper, we focus on the study of motion estimation and compensation in sign language video coding. We discuss and identify the approach of utilizing such motion information of sign language video for efficient sign language video communication applications. Key Words Motion Estimation, Motion Compensation, Video Coding, Sign Language Video Communication 1. Introduction Communication is the important means that people use in their correspondence to convey or exchange messages, news and information, thoughts and feelings. Good communication helps achieve better understanding and establish strong relationship. It enables people to live in the same society peacefully. Therefore, communication is crucially necessary component in human life as it is a fundamental for human survival and instrumental for people to achieve their objectives of working and living together in the society. However, for hearing-impaired, it is difficult for them to communicate with others to achieve the goals through the conventional telephone system due to their hearing disabilities, which become a barrier to their meaningful speaking. Presently, telecommunications technology has advanced rapidly but unfortunately it has been designed primarily for normal people, not for hearing-impaired. Thus, they are barred from achieving a successful life in the definition of Independent Living Philosophy [1]. Sign language is a visual language used by hearing-impaired people to communicate. A common device used by hearingimpaired people to communicate is the text telephone. Unfortunately, the speed of text conversation is limited by typing ability, and is at least 10 times slower than sign language [2]. Currently, there exist video conferencing tools such as Microsoft Netmeeting [3]. However, it is not designed to be easily used by hearing-impaired. There are several difficulties in sending video information over standard telephone lines. This is due to the fact that video signals contain much more information than other kinds of signals. Thus, two options exist; an either to send video through higher bandwidth medium or reduce the size of transmitted video data. Compression is mainly realized by exploiting the redundancy present in the data and is defined in several video coding standards H.263 [4], MPEG-4 [5]. The nature of sign language video is different from typical video sequences in a sense that the motion is compensated on sign s hand and arms. Thus, to apply video coding to sign language video, ones need to be able to identify signer s hands, arm, possibly face regions such that those regions can be coded with higher quality than other regions. Previous research in sign language video coding includes the identification of important regions of sign language video sequence images [6]. S. Akyol et.al [7] use image cues of color and motion for the fast detection of signing persons hands. In this paper, we focus on the study of motion estimation and compensation in sign language video coding. We discuss and identify the approach of utilizing such motion information of sign language video for efficient sign language video communication applications. The rest of the paper is organized as followed. In section 2, we briefly introduce the motion information in sign language video coding. In section 3, we describe

2 the experimental system and results. Then, the conclusions and future works is in section Motion Information in Sign Language Video Coding To exploit motion information in sign language video codec, one of the key elements is to motion estimation and motion compensation in video coding. A video sequence consists of a series of frames. To achieve compression, the temporal redundancy between adjacent frames can be exploited. That is, a frame is selected as a reference, and subsequent frames are predicted from the reference using a technique known as motion estimation. Motion Estimation(ME) is the estimation of the parameters of the model that describes the temporal variations, usually from consecutive frames. To encode differences between frames take the encoder needs to into account motion of the frames. The motion estimation is usually done on a block basis. A sequence of images contains an intrinsic, intuitive and simple idea of redundancy: two successive images are very similar. This simple concept is called temporal redundancy. The research of a proper scheme to exploit the temporal redundancy completely changes the scenario between compression of still pictures and sequence of images. The displacement of objects between successive frames is estimated (motion estimation). Hence the resulting motion information is exploited for an efficient interframe coding (motion compensation). Consequently the prediction error as well as the motion representation are transmitted instead of the frame itself. Interframe coding can be considered as a special case of predictive coding where the prediction is based on pixel values from the previous frame. For instance, in the portion of a scene with small motion, pixels are precisely predicted from the pixels at the same location in the previous frame. The last observation is not true any longer in the scenes with large motion. In this case, pixels in the previous frame spatially displaced by the appropriate motion vector are much more efficient for prediction. This prediction is named motion compensated(mc) prediction. The temporal redundancy can clearly be exploited to perform video compression. This has resulted in many motion based video compression strategies. Simple framedifferencing strategies assume the average motion is small, and simply compress the pixel-by-pixel difference between two frames. The motion estimation problem, in fact, consists of two related sub-problems: identify the moving object boundaries, so called motion segmentation estimate the motion parameters of each moving objects, so called motion estimation in strict sense Most of the motion estimation algorithms assume the following conditions: objects are rigid bodies; hence object deformation can be neglected for at least a few nearby frames objects move in a translational movement for, at least, a few frames illumination is spatially and temporally uniform; hence the observed object intensities are unchanged under movement occlusion of one object by another and uncovered background are neglected The purpose of motion estimation is very different depending on the application it is applied to. In image sequence analysis, the motion information is used to extract useful features of the image sequence. In image sequence interpolation and restoration, adaptive filtering exploits the motion information in order to avoid blurring of moving objects. Finally, motion information is used to reduce the temporal redundancies between successive frames in video coding. For each block(size NxN pixels) in the current frame, we need to search for best measures matching block in the reference frame. The block-matching algorithm(bma), the most commonly used in the standardized block-based coding schemes, searches the best match block of the current frame from the candidate blocks inside the search window(mxm pixels) with search range +/-d in the previous frame. The Full Search(FS) algorithm can find the optimal matching block by exhaustively checking all candidate blocks within the search window. However, its computational requirement is expensive. To overcome this problem, many fast blockmatching algorithms have been developed, such as 2-D logarithmic search(logs) [8], three-step search(tss) [9], new three-step search(ntss) [10], four-step search(fss) [11], new diamond search(ds) [12], blockbased gradient descent search(bbgds) [13], and etc. These fast algorithm use different search patterns and search strategies to find motion vector with much less number of search points compared to the FS algorithm. In this study, we perform experiments based on Full Search and TSS only. Distortion measures that are usually used to determine the best match are Mean Square Error(MSE) and Mean Absolute Error(MAE), as defined in eq. 1 and eq. 2, respectively. Figure 1 shows related parameters in block matching algorithm in motion estimation. (1) (2)

3 Figure 1 Related parameters in Block-matching algorithm in motion estimation a) original image b) frame difference 3. Experimental System and Results We use the standard sequence silent and a Thai Sign Language(TSL) sequence in this experiment. The format of silent sequence is QCIF(Quarter Common Intermediate) A videoconferencing standard that uses frames of 176 pixels by 144 lines at 30 frames/second. The format of TSL sequence is QCIF coded at 15 fps. The software tool is VCDEMO [14]. The input image sequence must be converted to yuv video format. Our experiment use full search and three step search technique. Motion is estimated on all frames of the sign language sequence. c) motion field d) motion compensated frame difference Figure 4 Motion information in terms of frame difference, motion compensate frame difference, and motion field of silent sequence. Frame 1 Frame 16 Frame 23 a) original image b) frame difference Figure 2 Example of video frames from silent sequence that demonstrates motion of sign language. c) motion field d) motion compensated frame difference Frame 3 Frame 39 Frame 162 Figure 3 Example of video frames from TSL sequence that demonstrates motion of sign language. Figure 5 Motion information in terms of frame difference, motion compensate frame difference, and motion field of TSL sequence.

4 a) Silent Sequence b) TSL Sequence Figure 6 Comparison between the variance of frame difference and motion compensation frame difference using full search for (a) silent (b) TSL sequences. a) Silent Sequence Figure 4 and 5 show the original image,motion field, frame difference and motion compensated frame difference for silent and TSL sequence respectively. The results indicate that the amount of information needed for coding and transmission of motion compensated frame difference is much lower than using frame difference. Thus, it is more efficient to utilize motion information to efficiently code sign language video. This conclusion is confirmed in a quantitative manner in figures 6 and 7. Figure 6 shows the comparison between the variance of frame difference and motion compensation frame difference using full search for silent and TSL sequences. The same settings are used for the sequences shown in figure 7 using three-step search. Since the variance of the frame difference represents the signal energy levels and the variance of the motion compensated frame difference represents the signal energy levels after motion compensation, therefore the variance can be used as an indicators of the number of bits used to encode that frame. The complexity of the block matching algorithm is reduced when using three-step search, while the accuracy is reduced, as can be seen from the increasing values of variance i.e., implied increasing number of bits used, as for motion compensated frame difference in figure 7, compared to full search algorithm in figure Conclusions and Future Works In this paper, we focus on the study of motion estimation and compensation in sign language video coding. The experimental results indicate using motion information in sign language video coding, i.e., the uses of motion estimation and motion compensation, can significantly reduce the bit rate because the motion of the sign language sequence is concentrated on well-defined region around the signer s hands and arms. Thus, the motion information can be used to indicate the region of interest and as an input to bit allocation scheme to allocate higher number of bits to improve the spatial and temporal qualities around the face, hands, and arms. Thus it results in the improvement of subjective quality of sign language video communication. Future works include the implementation and evaluation of priority and bit allocation scheme for sign language video coding. It is also necessary to optimize in complexity in terms of computation, storage, and memory requirements. 5. Acknowledgement b) TSL Sequence We thank the Ratchasuda foundation who supported the education and researches of the Thai sign language video communication. Figure 7 Comparison between the variance of frame difference and motion compensation frame difference using three-step search for (a) silent (b) TSL sequences.

5 References [1] L. Theeratorn, Behavior and satisfaction of people with hearing disabilities toward telecommunication technology, M.A. Thesis, Ratchasuda College, Mahidol University, 2002, ISBN [2] H. Nariman, Automatic Segmentation of the Face and Hands in Sign Language Video Sequences, Technical Report, Department of Electrical and Electronic Engineering, Adelaide University, SA 5005, Australia, July 3, [3] Microsoft Netmeeting, [4] ITU-T Draft Recommendation H.263, Video coding for low bit-rate communication, May [5] ISO/IEC JTC1/SC29/WG11. Doc. N4668, MPEG-4 Overview V.21. Jeju, March [6] Laura J. Muir, Video Telephony for the Deaf: Analysis and Development of an Optimised Video Compression Product, ACM Multimedia Conference, Juan Les Pins, France, 1-6 December [7] S. Akyol, P. Alvarado, Finding Relevant Image Content for mobile Sign Language Recognition, [8] J. Jain and A. Jain, Displacement measurement and its application in interframe image coding, IEEE Trans. Commun., vol. COMM-29, pp , Dec [9] T. Koga, K. Iinuma, A. Hirano, Y. Iijima and T. Ishiguro, Motion compensated interframe coding for video conferenceing, Proc.Nat. Telecommun. Conf., New Orleans, LA, Nov. 29-Dec , pp. G [10] S. Zhu and K. K. Ma, A new three-step search algorithm for block motion estimation, IEEE Trans.Circuits Syst.Video Technol., Vol. 4, pp , Aug [11] L. M. Po and W. C. Ma, A new diamond search algorithm for fast block motion estimation, IEEE Trans.Circuits Syst. Video Technol., Vol. 6, pp , June [12] S. Zhu and K. K. Ma, A new diamond search algorithm for fast block-matching motion estimation, IEEE Trans.Image Processing, vol. 9, pp , Feb [13] L. K. Liu and E. Feig, A block-based gradient descent search algorithm for block motion estimation in Video Coding, IEEE Trans.Circuits Syst.Video Technol., Vol. 6, pp , Aug [14] VcDemo Manual and Excercises, Delft University of Technology, Information and communication Theory Group.