a boundary-based method, their theory instead of relying on the shape of parts described by primitives is based on general principles underlying their

Transcription

1 PART{BASED DECOMPOSITION FROM EDGE IMAGES: HUMAN RESPONSES VS. MDL SEGMENTATION Maurizio Pilu Paul Frost Hewlett-Packard Research Laboratories Bristol BS34 8QZ, UK Abstract: This paper is concerned with an aspect of the structural properties of images of simple objects, namely their decomposition into generic parts using edge information only. In particular, it reports results of a psychological experiment conducted using experimentally naive subjects to assess their concept of part when presented with poor edge images. It compares the results with those of a recent computational method based on generic deformable part models and Minimum Description Length (MDL) segmentation of the scene in terms of these parts. The outcome shows a certain degree of convergence but disparity amongst subjects resemble diculties of computational approaches. 1 Introduction An essential part of the behavior of animals in general and people in particular is their ability to recognize objects. The fundamental concern of object recognition is how we move from the features revealed by a particular view of an object to a representation specifying the object. In computer vision, the concept of recognition based on parts has become popular. A representation of parts for recognition requires that they be invariant to transformations, unyielding in the presence of occlusions, stable regardless of viewing angle and have a hierarchical arrangement. Numerous theories have been put forward over the years that try to explain how humans perceive, decompose and recognize objects. For example, Marr and Nishihara [5] proposed a theory of axis-based representations. Biederman [1] has described complex objects as spatial arrangements of basic component parts, a description based on the combination of primitives such as generalized cones or geons. Whereas Homan and Richards [3] advocate

2 a boundary-based method, their theory instead of relying on the shape of parts described by primitives is based on general principles underlying their formations. However, Pentland [6] emphasizes that most complex natural shapes could be simplied as being composed of very simple parts and that these might be the basic components that we recover when analyzing images of natural objects. Most of these works employ, implicitely or explicitely, a sausage-like model of parts to speculate upon the ways computers should recover and deal with them. According to Pentland [6] these kind of models are \complex enough to be reliably recognizable, and yet simple enough to be used as building blocks for specic objects" Despite this well-established belief of what a part should be, an interesting question arose: Are computational results comparable to those produced by humans asked to segment objects into their composing parts from the same images? And do dierent people give same judgments? The easy answer to this question was that the results from humans would be probably very similar to those of [6] or [7, 8] and largely invariant across individuals. However, in order to more objectively investigate this claim, we devised a psychological experiment that would require experimentally naive subjects to judge what the composing parts of some simple objects were. Comparisons could then be made between the human interpretation and the part segmentation produced by the computational method in [7, 8] which is based on nding the Minimum Description Length (MDL) segmentation of edge images of objects in terms of deformable part models. 2 The Experimental Set-Up This section describes the set-up used to carry out the psychological experiment on a small group of 18 voluntary test subjects S#1... S#18. It is common practice for vision researchers to start with a concept or model of part when designing computational methods. For instance, in works such [6] or [7, 8] a blob or sausagelike model is used and [4] uses edge symmetries and non-accidental properties. In the psychological experiment that is going to be described, the test subjects are not explicitly told what kind of model to use nor what a part should be. In this way the scope of the experiment is broader than that of just nding the best combination of part models that t to the images. Here, subjects have to rst create a mental picture of what a part is in general and then apply it to other cases in the absence of an explicit denition. Hence, the experiment is about modeling and segmentation fused in an intuitive denition

3 Introduction This experiment aims at assessing how we segment objects into primitive \parts" (what a part is will be understood later) in the particular case where the visual input is an edge image. You may nd this experiment somewhat trivial, but it is important for validating some assumptions upon which most part-based recognition in computer vision is based. Procedure We have outlined an as-much-as-possible clue-less denition of what a part should be like: any kind of formal denition of part might have biased your opinion so we tried to avoid it. Here it is: I. For each test image do: II. Identify the separate objects most likely present in the image III. For each object do: IV. Sketch your best guess at the fewest and simplest parts it: {might be composed of or {might be made from or { most easily be broken into. In a few words, you have to rst identify the objects and then decompose the objects into what you think the constituting parts are according to the very intuitive denition given above. One thing you should not do is to reason too much about the images: the objects and the part separation should pop-up instinctively just at a quick glance. Whenever you see some clutter, you should try to grasp the main structure of the image, neglecting details. In the annexed gures (right)you can see some simple examples of what we mean with part segmentation. How to produce the results In the experiment you will be required to make some qualitative sketches of the parts. The parts should be described by their outlines and should be kept as much as possible in their relative position. Example image Example output sketch Figure 1: Guidelines given to each Subject in HTML Format (left). In addition, an example image was provided (top-left) along with a visual example on how to draw the part decomposition judgements (bottom-right). of what the deep nature of an object part might be 1). Having said this, however, from a pilot study with four experimentally naive subjects it became clear that the term \part" was too vague and ambiguous for the results to be meaningful. Thus, albeit avoiding any kind of technical or mathematical denitions, a general, slightly functional, denition of part segmentation was formulated: \the fewest and simplest elements an object a) might be composed of or b) might bemadefromorc) most easily be broken into". In addition, the provision of an example was deemed a necessary precursor to the experiment proper since the pilot study demonstrated the diculty subjects had carrying out the task. Figure 1 shows the actual experimental procedure that was handed to each of the subjects. 1) Powerful denitions such as Homan and Richards' transversality principle [3] were intentionally ruled out because subjects should have given intuitive replies.

4 I 1 I 2 I 3 I 4a & I 4b I 5 I 6 Figure 2: The six test images used in the experiment. I 1, I 2, I 4a b and I 6 are Canny edges from real images, whereas I 3 and I 5 were hand-drawn. Figure 2 shows each of the six test images. I 1, I 2, I 4a b and I 6 are Canny edges from real images, whereas I 3 and I 5 were hand-drawn. I 1 is a hand I 2 is a telephone handset I 3 is a synthetic image of a hammer on the top of a bottle and a ovoidal object I 4a (top cluster of edges) is a marker beneath a screwdriver whereas I 4b (bottom cluster of edges) is a wooden stick I 5 is a hand-drawn tree I 6 is a toy rabbit. Note that image I 3 and I 4a are of particular interest because subjects had to work out the overlap and possibly identify occluded objects as a whole. While subjects received both the procedure and the test images in HTML format, they were required to return their responses in paper format. 3 Experimental Results Once the experimental data was gathered, the responses of all subjects for each test image were segregated into classes. New classes were formed whenever a response diered both in terms of the number of parts and their relative positioning to others. Variations in the parts' dimensions and/or shape did not warrant a new class of object as these variations were attributed to dierent drawing ability and/or care invested in the experiment by subjects. It should be borne in mind that although the division of I 4 into two sub-images was merely for the purpose of analysis, subjects were presented the whole image as we shall see later,

5 this biased the interpretation of both I 4a and I 4b. Figure 3 shows hand drawings of the classes of responses that were obtained from all subjects. Each class is tagged by a capital letter and individual parts for each class are numbered progressively and will be indicated as class/part#. In those cases where a subject failed to produce an answer complying to the guidelines (see the Procedure) a Null entry was made. Table 1(left) shows the individual class responses made by each subject for each of the test images. Frequency counts for class responses partitioned by image are shown in Table 1(right). The disparity and correspondence between subjects' responses for each of the images is shown clearly in Table 1(left). Refer to Figure 3 for tag letters to the classied responses. For I 1,ClassBwas the most popular rendering. Perhaps Class C should have been considered Null since one of the subjects used their higher level knowledge of the hands anatomy or structure to decompose the object into the fewest number of parts. Class B was the most popular rendering for I 2 and accounted for 50% of the responses. However, Classes A, C, & D seem to reect more accurate renderings in terms of the requirements of the task. Indeed they are more similar to the results of the MDL algorithm [7, 8] depicted in Figure 4. Class A was overwhelmingly the most frequently chosen rendering (89% of subjects) for I 3. One subject however, saw B/2 and B/3 as disjoint another saw the hammerhead as composed of two parts (C/5 and C/6) and also the bottleneck and body of the bottle as separate (C/2). The cluster I 4a received a peculiar interpretation by most subjects, who seemed to draw an airplane with a banner or smoke trailing. Class B was the most popular response in which the two alleged wings were considered to be two separate parts. In A and D the shaft of the screwdriver was not reported, maybe due to its thinness. Class C actually reects the original image but was only rendered by 3 subjects. Results are quite interesting for I 4b too. The four test subjects that decided for the singlepart class A also had the \airplane" interpretation of I 4a (they were asked later about their choice) because they thought itwas a cloud, revealing the strong inuence of high-level interpretation. Overall, ignoring details, the results are equally good interpretations diering only in where and how big the main body is. In the case of I 5 all subjects responded in the same fashion. Either the image of the tree was signicantly easier for the subjects to carry out the task of part segmentation or alternatively

6 INDIVIDUAL RESPONSES I1 I2 I3 I4a I4b I5 I6 S#1 E C A B B A C S#2 A B A C C A I S#3 B B A B C A H S#4 B B B B F A I S#5 B B A B A A D S#6 B D A D A A - S#7 B B A B A A G S#8 C C C F G A F S#9 B A A D D A B S #10 B D A B C A D S #11 A B A C D A E S #12 B B A B C A J S #13 B - A B B A C S #14 B B A B A A B S #15 B E A A C A A S #16 B B A C C A I S #17 B D A B G A J S #18 A C A B E A C CLASSES COUNT I1 I2 I3 I4a I4b I5 I6 A B C D E F G H I J Table 1: Left: Individual responses of subjects in terms of the classes depicted in Figure 3 Right: Classes count for each image. The dash \-" indicates a null reply. a higher level knowledge of a tree's structure played a facilitatory role. For I 6 classication proved more dicult probably due to the complexity of the original image. In all responses the head and ears were clearly identied. Curiously, the nose was not always reported 4 Discussion & Comparisons with Computational Method In this section we will endeavor to make comparisons between the data collected by the psychological experiment just described and the results obtained by the computational method presented in [7, 8] based on the Minimum Description Length interpretation of the edge images in terms of deformable \sausage-like" models. Figure 4 shows the part segmentation results that will be used for comparisons. For I 1 the MDL results are comparable to classes A or B in Figure 3. Since classes A and B account for 16 out of 18 responses, this can be seen as a good result. Notice that the back of the hand could not be recovered by the MDL method because the correct hypothesis was not generated [7]. In the case of the handset in I 2,theMDLresults are in harmony with those represented by classes A, B and D of Figure 3 but as in the previous case, they do not match any of them precisely. Rather, the MDL output has three parts of class B (B/1, B/2 and B/3) but also the \covers" as in A/3, D/1 and D/5. The spurious interpretation of the handle appears

7 Figure 3: Hand-sketches of the part decomposition of each of the six test images, partitioned into dierent response classes. also in class C, albeit dierently. Classes A, B and D account for 13 of the 17 valid responses and therefore this MDL result can be considered positive too. For the composite I 3 where actual objects were overlapping, the (very stable) MDL result

8 Figure 4: Results produced by the Minimum Description Length method of [7, 8] is precisely that of class A in Figure 3, which accounts for 16 responses out of 18. By any standards, therefore, the computational method agreed with human judgments. As said before, for I 4a the results are not so exciting...for the test subjects. However, the MDL algorithm, on the other hand, saw the one-part interpretation as cheaper than the two-part one. The one-object interpretation can be considered the correct one and had not the test subjects reasoned too much about the meaning of that poor-quality image, perhaps they would have given the correct, more low-level, one-model interpretation. As previously mentioned the results for I 4b have been aected by two factors: a) scenario imagined and b) the inherently arbitrary decomposition. In the case of the MDL output, the result has been aected by scale problems [7]. Due to this, hypotheses like B and F in Figure 3 could not possibly happen. Obviously, because of the nature of the models employed, also classes A and D could not occur but that is not a big problem because they can be considered as failed decomposition by \too holistic" subjects. By considering classes C and E together (they just dier for the little tail part), the MDL output is in accordance with 8 out of 18 answers. The tree in I 5 has received a single interpretation by all subjects, which is shown in Figure 3. In the MDL output the middle and right little branches have not been selected again for scale reasons if this detail is ignored, the results can be considered to be in perfect correspondence.

9 Finally, there is the messy case of the toy rabbit (I 6 ). The interpretation produced by the MDL method is quite simple, as can be seen in Figure 4, but was the result of scale problems, rather than cleverness. If we neglect the ill-dened lower body of the toy rabbit, it turns out that ears and head, the most prominent entities, have been perceived by 16 out of 17 subjects, as seen in Figure 3. These three parts also appear stably in the MDL output. For the lower part of the body the test subjects have given a plethora of dierent interpretations but the MDL performed no better. 5 Conclusion Summing up, overall the part segmentation produced computational methods such as [6] or [7, 8] do produce results in accordance to those given by many of the test subjects. The test objects used in the test scenes were of limited complexity because of the inherent limitation of part-based representation schemes and for some other limits and pitfalls in the specic techniques used [7], but the test subjects were not told what kind of model to use in their decomposition so the experiment can be considered fair from this point of view. Despite the high correspondence that has been shown to exist between the computed and perceived parts, dierences are also evidenced: disparity between the computed and perceived parts is apparent in all instances with the exception of the image of the tree. But perhaps an important lesson (or conrmation, depending upon one's stand) is that the diculties and dierences in interpretation between subjects casts doubts as to whether structure-based but context-free part segmentation is founded and if the edges' primary role in categorization into parts [9, 2] is entirely correct. Since it is evidenced that higher order knowledge has been recruited by subjects to solve the task, some instances of future experiments might consider the use of abstract shapes. For instance, in segmenting the hand in I 1 a knowledge of the hands' structure inuenced the number of parts reproduced in some cases when clearly these cues were absent from the original test image, though this cannot be seen as unexpected since this was inherent in the design of the experiment (i.e. "..you have to rst identify the objects..."). Evidently, we are still some way from an all-encompassing theory of object recognition which can integrate and account for the ndings from computational modeling. But overall, we feel that the main conclusion (and original aim) of the experiment was to conrm that part decomposition is not such an under-specied task as is often thought. In particular, approaches based on the Occam`s Razor such as MDL do in fact yields realistic interpretations, even in their multi-stability behavior where a number of interpretations of the data are equally acceptable.

10 Acknowledgments The authors acknowledge the suggestions of Dr. Robert B. Fisher and Dr. Chris Malcolm (University of Edinburgh) and Prof. Chris Taylor (Universiy of Manchester) and thank all the voluntary test subjects. References [1] I. Biederman. Recognition-by-components: A theory of human image understanding. Psychological Review, 94:115{147, [2] T.O. Binford. Visual perception by computer. In Proceedings of the IEEE System Science and Cybernetic Conference, Miami, December [3] D. Homan and W. Richards. Parts of recognition. In A. Pentland, editor, From Pixels to Predicates. Ablex, Norwood, NJ, [4] D. Lowe. Perceptual Organization and Visual Recognition. Kluwer Academic Publishers, Boston, MA, [5] D. Marr and H.K. Nishihara. Representation and recognition of the spatial organization of threedimensional shapes. Proceedings of the Royal Society of London, Series B, (200):269{294, [6] A.P. Pentland. Perceptual organization and the representation of natural form. Articial Intelligence, 28:293{331, [7] M. Pilu. Part-based Grouping and Recogntion: A Model-Guided Approach. PhD Thesis, Department of Articial Intelligence, University of Edinburgh, Scotland, August [8] M. Pilu and R.B. Fisher. Part segmentation from 2D edge images by the MDL criterion. Image and Vision Computing, 15(8):563{573, August [9] M. Wertheimer. Laws of organization in perceptual form. In W.D. Ellis, editor, A Source Book of Gestalt Psychology. Harcourt Brace, New York, 1923.