Supplemental Materials for Paper 113

Size: px

Start display at page:

Download "Supplemental Materials for Paper 113"

Darcy Chase
5 years ago
Views:

1 Supplemental Materials for Paper 113 August 19, 2012 In this document, we include the following items: 1. Concept definition: the full list of our 101 concepts including actions, scenes, and objects. 2. Novel events recognition: we first describe a set of relevant concepts for each event, which is used to define the event model in our paper for recognizing novel events without training examples. Then we show a concept similarity matrix, which can be used to enhance the human defined event models. In the experimental part, we present more results in the form of DET curve [2] for each event. From these curves, we can easily compare our approach with the baselines. Finally, we present some preliminary results from a set of very challenging experiments. In these experiments, we define and train the concept detectors on a third-party data set, rather than the TRECVID MED11 [1] data set which is our event video data set. Our goal is to show it is feasible to train general concept detectors from a third-party data set. 3. Event recounting: more demonstration and experimental results are demonstrated for event recounting. 1

2 Figure 1: The full list of concepts defined based on the TRECVID MED11 data set. 1 Concept Definition Based on the TRECVID MED11 data set [1], we define 101 concepts including actions, scenes, and objects, as shown in Fig. 1 2 Novel Event Recognition 2.1 Event definition in terms of concepts As we discussed in the paper section 3.4, in order to recognize a novel event, we need to construct an event model, which is a vector with each bin value indicating the presence/absence of the corresponding concept in the event. We define the relevant concepts for each event as follows. E01 Attempting a board trick: Person falling, person flipping, person jumping, person running, person sliding, person surfing, person walking, person landing, standing on the board. E02 Feeding an animal: Animal approaching, Animal eating, Hands visible, Person petting, Person Pouring, Person walking, Person throwing, Animal grabs things. E03 Landing a fish: Hands visible, Person bending, Person casting, Person reeling, person walking. 2

3 E04 Wedding ceremony: Person dancing, people dancing, Person hugging, Person kissing, Person taking pictures, Person clapping, Person laughing, Person walking, Bride, groom, priest standing, Walking down the aisle, Person cheering, person singing. E05 Woodworking: Hand visible, Person carving, Person cutting, Person drilling, Person hammering, Person planning, Person sawing, Person turning wrench, Person writing, Person cutting fabric. E06 Birthday party: Person bending, Person blowing candles, Person cheering, Person Clapping, Person dancing, Person drinking, Person eating, Person jumping, Person kissing, Person walking, Person hugging, Person laughing, Person rolling, Person singing, Person taking pictures, Person waving. E07 Changing a vehicle tire: Hand visible, Person fitting bolts, Person hammering, Person jacking car, Person squatting, Person turning wrench, Person using tire tube, removing hubcap wheel, taking a tire out, turning wrench unscrewing. E08 Flash mob gathering: People dancing, Person dancing, Person bending, Person clapping, Person flipping, Person falling, Person jumping, Person running, Person squatting, Person taking pictures, Person walking, Dancing and Singing, Moving in a coordinated fashion. E09 Getting a vehicle unstuck: Person bending, Person pushing, Person walking, Vehicle moving, Wheel rotating. E10 Grooming an animal: Hands visible, Person cutting, Person petting, Person bending, Person washing, Person pouring, Animal eating. E11 Making a sandwich: Hands visible, Open door, Person eating, Person pouring, Person using knife, Spreading scream, Person cutting, Cutting dishing up fillings, Placing fillings on bread, Slicing and cutting, Spreading condiments, Toasting bread, Person carving. E12 Parade: People marching, Person dancing, Person running, Person walking, Person waving, Vehicle moving, Person clapping, People dancing, Person cheering. E13 Parkour: Person jumping, Person climbing, Person falling, Person flipping, Person rolling, Person running, Person sliding, Person walking, Jumping over obstacles, Scaling walls trees. E14 Repairing an appliance: Hands visible, Open door, Person fitting bolts, Person hammering, Person squatting. E15 Sewing project: Hands visible, Person bending, Person cutting, Person pointing, Person cutting fabric, Person sewing, Person writing, Person planing, Person carving. 3

Figure 2: An action concept similarity matrix. Colder color means less similarity. 2.2 Experimental results We define the event model using action concepts.

4 Figure 2: An action concept similarity matrix. Colder color means less similarity. 2.2 Experimental results We define the event model using action concepts. In other words, each event model vector is a 81-dimensional binary vector with each bin indicating the presence/absence of the corresponding concept in the event. As we know, this definition may be subjective and noisy. Se we enhance the manually defined event model using the concept similarity as described in Sec We estimated the concept similarity matrix from a training data set excluding videos from the novel event. Fig. 2 shows one similarity matrix. In the paper, we show the comparison in terms of Average Precision between various zero-shot learning approach in Fig. 8. Here, we present more experimental results in terms of DET curves [2] in Fig 4, where the X and Y axis represent Miss Detection rate ( number of false negative number of false positive total number of positive ) and False Alarm rate ( total number of negative ) respectively, and Blue, Green and Red curves represent approach LLFeat-05 (training based with 5 positive training examples using low-level features), zero-shot:binary (recognizing novel events using human-defined event models), and zero-shot:ss-based (enhancing human-defined event models using concept similarity) respectively. We notice that about 9 over 12 events obtained better performance by enhancing the event models with concept-to-concept similarity. We also conducted more challenging experiments, in which we define event models using a subset (i.e., 117 concepts) of the 346 concepts defined for TRECVID Semantic Indexing task ( http : //www nlpir.nist.gov/projects/tv2011/tv2011.html#sin), 4

5 Figure 3: Preliminary results (Green curves) for novel event recognition using 117 concept detectors trained on a third-party data set. For comparison, we also present the results (Blue curves) of training based approach with 2 positive training examples using low-level features. We can see our zero-shot learning achieved competitive results with training based approach using 2 positive training examples. rather than our 81 action concepts. In other words, all the concept detectors are trained on TRECVID IACC data set (http : //www nlpir.nist.gov/projects/tv2011/tv2011.html#iacc.1.a), which is different TRECVID MED11 data set. Therefore, we believe these concept detectors are more general. We define five events using the 117 concepts. For example, we associate Event Attempting a board trick with concepts like Beach, Building, Cityscape, Male person, Legs, Oceans, Walking, Skating, and so on. Please check the included Execel file event definition by SIN concepts.xlsx for the detailed definition in terms of 117 concepts. These definitions directly come from the existing knowledge of one person to the events. Fig. 3 shows the preliminary results (Green curves) on Event 01 to 05. We compare the results to the training based recognition with 2 positive training examples using low-level features as we describe in the paper. We can see, the zero-shot learning achieves better results for 4 of the five events than the training based approach. These experiments demonstrate the effectiveness of our approach for recognizing novel events when the concept detectors are trained on a third-party data set. 5

6 3 Event Recounting In the paper, we show two examples of event recounting in Fig. 10. Here, we show two more examples of event recounting on event 03 Landing a fish and Event 13 Parkour in Fig. 5. In addition, Fig. 6 shows the top recounted concepts all 15 events. The first and second column are the event ID and the event name respectively. The third column lists the top six recounted concepts in a descending order. For example, in event 001, standing on the board is the most recounted concept than the other five ones. As we can see, most of the top recounted concepts are quite related to the event. One counter example is in event 015, the second top recounted concept is placing fillings on bread. It is worth nothing that all of our concept detectors focus more on motion features than static features. References [1] [2] A. F. Martin, G. Doggington, T. Kamm, M. Ordowski, and M. Przybocki. The det curve in assessment of detection task performance. Proc. Eurospeech, 4: ,

7 Figure 4: DET curves [2] (Being closer to the Origin point is better) for 12 events. The Blue, Green and Red curves represent approaches LLFeat-05 (training based with 5 positive training examples using low-level features), zero-shot:binary (recognizing novel events using human-defined event models), and zero-shot:ss-based (enhancing human-defined event models using concept similarity) respectively. About 9 over 12 events show improved performance by enhancing the event models using concept-toconcept similarity. 7

Figure 5: Recounting examples. Given a video clip that is classified positively by the event classifier (Event 03 and Event13), the top six recounted concepts are shown in (a)(c).

8 Figure 5: Recounting examples. Given a video clip that is classified positively by the event classifier (Event 03 and Event13), the top six recounted concepts are shown in (a)(c). The center frame (shown in a yellow bounding box) of the sliding window with the maximum action concept confidence is used as the action concept exemplar. The red bounding box shows the detected object concept. In (b)/(d), we show a list of the most recounted concepts among all positive videos of Event 03/ Event 13. The x axis of the histogram shows the indices of the concepts and y axis of histogram shows the number of positive videos in that of a concept ranks top. The list of concepts is in the same of order that the arrow line intersects the histogram bars. 8

9 Figure 6: Top recounted concepts for all 15 events. The first and second column are the event ID and the event name respectively. The third column lists the top five to six recounted concepts in a descending order. 9

Using the Soundtrack to Classify Videos

Using the Soundtrack to Classify Videos Dan Ellis Laboratory for Recognition and Organization of Speech and Audio Dept. Electrical Eng., Columbia Univ., NY USA dpwe@ee.columbia.edu http://labrosa.ee.columbia.edu/