2018 Keypoints Challenge

Transcription

1 2018 Keypoints Challenge Matteo Ruggero Ronchi COCO and Mapillary Joint Visual Recognition Challenges Workshop Sunday, September 9th, Munich, Germany

2 COCO Dataset

3 COCO Keypoints Dataset (I) Multiple Perspectives, Instances, Sizes, Occlusions: Overall Statistics (train/val): 17 types of keypoints. 58,945 images. 156,165 annotated people. 1,710,498 total keypoints. 3 / 24

4 COCO Keypoints Dataset (II) Multi-Instance Dataset: Avg of ~2 annotated people per image. Up to 13 annotated people per image. Number of images Number of annotations 4 / 24

5 COCO Keypoints Dataset (III) Distribution of the number of keypoints: 70,000 Number of instances 60,000 50,000 40,000 30,000 20,000 10, Number of keypoints 5 / 24

6 Evaluating Keypoint Predictions How to measure localization accuracy: Bounding Box IoU Mask IoU Object Keypoint Similarity 6 / 24

7 Keypoints Evaluation Metric Object Keypoint Similarity (OKS): 7 / 24

8 COCO Keypoints Task Simultaneous detection and keypoint estimation: 1 OKS = 0.5 OKS = precision recall recall OKS.5 OKS.95 Ground Truth 8 / 24

9 2018 Keypoints Challenge COCO Challenge / 24

10 2018 Keypoints Challenge Leaderboard (I) (*) Ensemble model method (+) Used external keypoints training dataset (=) Mask RCNN based 80% 70% 12 teams in ~5% AP +4.5% absolute ~6% relative 60% 50% 40% 30% 20% 10% 0% Megvii (+*) MSRA (+*) Sea Monsters (+*=) DGDBQ (+*=) KPLab (+*=) ByteDance-SEU (*=) SNU CVLAB (=) fadivugibs (*=) Megvii [17] jd_y ByteCV Fast-20-FPS METU liuxhy237 Raven-DL TFMAN Huya ustclijia UOU_ISLab rovio 10 / 24

11 Absolute Winner Improvement (I) 100% COCO AP (20%) (15%) (8%) 75% 50% 60.5% 72.1% 76.6% 25% 0% / 24

12 2018 Keypoints Challenge Leaderboard (II) Better performance at looser localization thresholds: 100% 90% 80% 70% 60% 50% 40% 30% 20% 10% AP 50 AP 75 ~16% above COCO AP 0% Megvii MSRA Sea Monsters DGBQ KPLab ByteDance-SEU SNU CVLAB fadivugibs Megvii [17] 12 / 24

13 Absolute Winner Improvement (II) Breakdown over OKS Threshold 95% Winner 2017 Winner % 90% 85% + 1.6% 80% OKS 50 OKS / 24

14 2018 Keypoints Challenge Leaderboard (III) Instance scale is an important factor: 90% 80% 70% Large Instances Medium Instances 60% 50% 40% 30% 20% 10% 0% Megvii MSRA Sea Monsters DGBQ KPLab ByteDance-SEU SNU CVLAB fadivugibs Megvii [17] 14 / 24

15 Absolute Winner Improvement (III) Breakdown over Instance Size 85% 80% Winner 2017 Winner % 75% 70% + 1.9% 65% 60% Medium Large 15 / 24

16 Performance Breakdown over Keypoints COCO AP varies across keypoints 90% 80% Megvii MSRA Sea Monsters DGBQ KPLab ByteDance-SEU SNU CVLAB fadivugibs 79.4% avg COCO AP ~3% spread 70% 60% 72.8% avg COCO AP ~4% spread 50% all face upper torso lower limbs 16 / 24

17 A Closer Look at Errors Taxonomy of Errors for Multi-Instance Pose Estimation: JITTER INVERSION SWAP MISS SCORING [1] Ronchi et al, Benchmarking and Error Diagnosis in Multi-Instance Pose Estimation, ICCV / 24

18 A Closer Look at Errors [1] Ronchi et al, Benchmarking and Error Diagnosis in Multi-Instance Pose Estimation, ICCV / 24

19 Precision A Closer Look at Errors (II) Fine-grained Precision-Recall Curves Team Megvii 1 OKS 0.5 =.932 OKS 0.75 =.828 OKS 0.95 = Recall 1 Localization Errors: Jitter: Inversions:.962 Swaps:.960 Misses: Scoring & Bckgd. Errors:.962 False Neg.: False Pos.:.980 Opt. Score: Recall 1 Localization Errors: Jitter: Inversions:.957 Swaps:.937 Misses: Scoring & Bckgd. Errors:.962 False Neg.: 1.00 False Pos.:.980 Opt. Score:.980 [1] Ronchi et al, Benchmarking and Error Diagnosis in Multi-Instance Pose Estimation, ICCV Recall 1 Localization Errors: Jitter: Inversions:.346 Swaps:.311 Misses: Scoring & Bckgd. Errors:.627 False Neg.: False Pos.:.703 Opt. Score: / 24

20 A Closer Look at Errors (II) Fine-grained Precision-Recall Curves Team Megvii False Neg. False Pos. Opt. Score Jitter Inversions Swaps Misses [1] Ronchi et al, Benchmarking and Error Diagnosis in Multi-Instance Pose Estimation, ICCV / 24

21 Localization Errors Best performance for each type of Localization Error Good Nose Shoulders Hips Inversion Swap Eyes Elbows Knees Jitter 5% 1% 3% 10% 82% Good Megvii (82.1%) Miss 5% 11% 8% 12% 10% 9% 19% 16% Miss 10% Megvii (4.6%) MSRA 9% 11% 15% Ears 5% 3% 14% 16% Swap 5% 22% MSRA (0.9%) liuxhy237 METU Wrists 19% 17% 6% 24% 10% Inversion Ankles 4% 14% 7% Megvii (2.6%) 7% 8% 9% 14% 16% 15% 10% 11% 10% Jitter Megvii (9.7%) 21 / 24

22 Occlusion and Crowding Benchmarks (I) COCO Benchmarks of image complexity: Occlusion: number of visible keypoints Crowding: number of overlapping instances (IoU > 0.1) 2 [1, 5] 2 [6, 10] N. of Keypoints 2 [11, 15] 2 [16, 17] 0 N. of Keypoints 2 [1, 5] 2 [6, 10] 2 [11, 15] 2 [16, 17] N. of Overlaps 2 [1, 2] 0 N. of Overlaps 2 [1, 2] / 24

23 Occlusion and Crowding Benchmarks (II) Overall challenge performance is saturated by the easiest benchmarks: 1 2 [1, 5] 2 [6, 10] N. of Keypoints 2 [11, 15] 2 [16, 17 0 Precision N. of Overlaps 2 [1, 2] 0 1 Precision Precision 0 0 Recall 1 0 Recall 1 0 Recall 1 0 Recall 1 Team Megvii 2 [1, 5] 2 [6, 10] N. of Keypoints 2 [11, 15] 2 [16, 17] 23 / 24

24 Summary of Findings 2018 Keypoint Challenge Take-aways: Relative AP improvement of the winner is slowing down. Majority of the top methods are based on similar approaches and the performance gap is very small. Limbs (arms and legs) are the hardest to correctly detect. Performance is saturated by the easiest benchmarks. We need to broaden current benchmarks with challenging images (high occlusion / low number of keypoints). 24 / 24

25 2018 COCO Keypoints Challenge Team Megvii Face++ MSRA Sea Monsters G-RMI Position 1 st 2 nd 3 rd 4 th Invited Speakers: Team Megvii / (heard that!) Team MSRA / (10:15am - 10:30am) FAIR SJTU 5 th 6 th Samsung-pose 7 th 25 / 24