Effect of Calibration Data on Forensic Likelihood Ratio from a Face Recognition System

Transcription

1 Effect of Calibration Data on Forensic Likelihood Ratio from a Face Recognition System Tauseef Ali, Luuk Spreeuwers, Raymond Veldhuis Biometric Pattern Recognition Group, FEEMCS, University of Twente PO Box 7, 75 AE Enschede, the Netherlands {T.Ali, L.J.Spreeuwers, R.N.J. Veldhuis}@utwente.nl Didier Meuwly Netherlands Forensic Institute PO Box, 9 AA The Hague, the Netherlands D.Meuwly@nfi.minvenj.nl Abstract A biometric system used for forensic evaluation requires a conversion of the score to a. A likelihood ratio can be computed as the ratio of the probability of a score given the prosecution hypothesis is true and the probability of a score given the defense hypothesis is true. In this paper we study two different approaches of a forensic computation in the context of forensic face recognition. These approaches differ in the databases they use to obtain the score distribution under the prosecution and the defense hypothesis and therefore consider slightly different interpretation of these hypotheses. The goal of this study is to quantify the effect of these approaches on the resultant in the context of evidence evaluation from a face recognition system. A state-of-the art commercial face recognition system is employed for facial images comparison and computation of scores. A simple forensic case is simulated by randomly selecting a small subset from the FRGC database. Images in this subset are used to estimate the score distribution under the prosecution and the defense hypothesis and the effect of different approaches of a computation is demonstrated and explained. It is observed that there is a significant variation in the resultant s given the databases which are used to model the prosecution and defense hypothesis are varied.. Introduction A score obtained from a face recognition system quantifies the similarity between the pair of input images while taking into account their typicality. In biometric applications such as access control to a building and e-passport gates at some airports, we choose a threshold from the range of the score and consequently any score above the threshold implies a positive decision and vice versa []. However, in a criminal case, there are facial images from a crime scene, e.g., facial recordings from a surveillance camera as well as images from the suspect. The responsibility of a forensic scientist is to give a Likelihood Ratio (LR) instead of a decision to the court []. Then, it is the responsibility of the judge or the jury to make a decision which involves other sources of information about the case at hand. Use of a LR to report the output of a biometric comparison is gradually becoming a standard way of evidence evaluation from score-based biometric systems. A LR is a more objective and useful output in forensic evaluation than simply a score []. A general description of the LR framework for evidence evaluation from biometric systems can be found in [, ]. It is applied to several biometric modalities including forensic voice [5], speaker [6, 7, 8] and fingerprint comparison [9]. Preliminary results of evidence evaluation using this framework in the context of face and handwriting recognition systems are presented in [,, ]. A LR is the probability of the score given the prosecution hypothesis is true divided by the probability of the score given the defense hypothesis is true: LR(s) = P (s H p) P (s H d ) where s, considered as the evidence, is the score obtained by comparison of the image from the suspect with the image found at the crime scene. H p and H d are two ()

2 mutually exclusive and exhaustive source-level hypotheses defined as follows: H p : The pair of input images that produced score s originated from the same source. H d : The pair of input images that produced score s originated from different sources. The LR computes a conditional probability of observing a particular value of the evidence s with respect to H p and H d. It is therefore an empirical tool to evaluate and compare these two hypotheses concerning the likely source of the trace image found at the crime scene. Once a forensic scientist has computed the LR, it can be interpreted as a multiplicative factor which updates the prior odds (before observing the evidence from a biometric system) to the posterior odds (after observing the evidence from a biometric system) using the Bayesian framework: P (H p s) P (H d s) = P (s H p) P (s H d ) P (H p) () P (H d ) In this framework, the judge or the jury is responsible for quantification of the prior beliefs about H p and H d while the forensic scientist is responsible for the quantitative evaluation of the evidence s in the form of a LR. The hypotheses H p and H d can be specifically interpreted in a slightly different way so that the same-source and different-source condition is linked with the specific suspect in a forensic case. These different interpretations correspond to differences in the pairs of images used to obtain the score distribution under H p and H d. In forensic evaluation, these pairs of images are called calibration data. The purpose of this paper is to quantify the evidential value from facial images using the framework and to quantify the effect of different calibration data used to obtain the score distribution under the prosecution and defense hypothesis []. The study is carried out in the specific context of face recognition, however, the concepts and proecedure described apply to any biometric system which computes a score for an input pair of samples. The paper is organised as follows. In section we briefly review the LR computation process from biometric scores and discuss the employed score-to-lr conversion method. Section discusses the two different approaches and the differences in the interpretation of the hypotheses it implies. Section reviews some existing work which studies the effects of different calibration data on LRs in the context of forensic speaker and handwriting recognition and presents the comparison procedures. Section 5 explains experimental setup and the degradation process to obtain trace-like images. Section 6 presents results by mapping score-axis to Log LR (LLR) using the two different approaches of LR computation. Finally section 7 draws conclusions and points toward future research directions.. Computation of a LR.. Computation of calibration scores -based biometric systems output two classes of scores. The first one is the result of the comparison of two samples produced by the same source. When comparing a set of samples produced by the same source, there is some variation in the score values output by a biometric system. Each modality has different nature of variations in the samples produced by the same source, for example, in case of face recognition systems it is caused by lighting condition, facial expressions and partial occlusion of the face, etc. A set of scores obtained by comparing samples from the same source represent the within-source variability of the score and is referred to as the within-source scores. Similarly, comparing a set of samples produced by different sources results in a set of scores that represent the between-source variability of the score and is referred to as the betweensource scores (see Fig. ). s in the within-source and in the between-source sets are collectively called calibration scores where the pair of samples to obtain these calibration scores are referred to as calibration data. W pairs of samples from same source B pairs of samples from different sources Face Recognition System Face Recognition System Within-source scores {s,s,..,sw} Between-source scores {s,s,..,sb} The pair of samples consisting of the suspect and the trace Likelihood ratio computation s Face Recognition System Figure : Computation of a score-based LR LR (s) In order to compare a pair of input images and obtain a score, we use a state-of-the-art commercial face recognition system []. This system computes a score value in the range of and... Mapping score-axis to s using calibration scores -based LR computation can be considered as a mapping function from score to LR. Given a set of calibration scores, there are several methods to map the score-axis to Likelihood Ratios (LRs). These methods as described in [5] can be classified as parametric or non-parametric. When the distribution of the within-source and the betweensource scores sets are similar to a standard probability density function (pdf), the pdf of score under H p and H d can be estimated by fitting standard pdfs with certain parameters using maximum likelihood estimation to these sets of

3 scores [9]. Another possible parametric approach is to estimate the ratio of the pdf under H p and pdf under H d using logistic regression [5]. In nonparametric category, there are histogram binning, Kernel Density Estimation (KDE) and finding slope of Receiver Operating Characteristic Convex Hull (ROCCH) [6]. For the scores obtained from the face recognition system in this study, we propose the use of ROCCH procedure due to the variability in the distribution of scores obtained and the mismatch of the distribution of scores to any standard family of pdf. This approach is preferred because it can ensure that the resultant variation in LRs are due to the differences in the within-source and between-source sets and not due to the poor fitting of the densities to the calibration score sets. Once ROCCH is computed for a given set of calibration scores, LR for a given score is the slope of the corresponding segment of the ROCCH and can be computed as follows: LR(s) = W B P (H d) () P (H p ) where W and B are the number of the within-source and the between-source scores respectively in the corresponding segment of the ROCCH on which score s lies. The value P (H d) P (H p) is computed from the size of the within-source and the between-source sets. It is interesting to note that computing ROCCH is equivalent to computing ROC of the posterior probabilities obtained by Pool Adjacent Violators (PAV) algorithm [6]. This argument leads to an alternative way of implementation; computing posterior probabilities using PAV and then plugging it into Bayesian formula along to compute LRs. Once the ROCCH procedure is applied to compute LRs, there are a group of scores for which the posterior probabilities are either or. The log odds of these posterior probabities results in minus infinities and plus infinities respectively. To avoid this problem, a procedure similar to [7] is followed. We replace a score in the between-source set by the maximum score in the within-source set and a score in the within-source set by the minimum score of the between-source set. These replaced scores can be considered to represent scores which were not encountered in the calibration scores because there is not enough calibration data, but which could have occurred. with P (H d) P (H p). Suspect-anchored and suspect-independent calibration data Based on the quantity of the available data from the suspect, the within-source and the between-source scores can be either anchored to the suspect or it can be general withinsource and between-source matches using all suspects from the potential population database. Suspect-anchored approach To compute the within-source scores, a set of images from the suspect can be compared with another set of images from the suspect. This is referred to as suspectanchored approach [8]. The two sets of images are referred to as suspect-reference and suspect-control data sets. For better calibration, the images in the suspect-control data set should be as close as possible to the trace and the images in the suspect-reference data set should be as close as possible to the potential population database. Cross comparison of all images in the suspect-reference and suspect-control data set results in a set of scores that can be used to model the distribution of scores under the prosecution hypothesis. Similarly, for modelling the distribution of scores under the defense hypothesis, images in the suspect-control data set are compared with images in the suspect-reference data set of all the suspects in the potential population [8]. The suspect-anchored approach implies considering the following interpretations of the prosecution and defense hypotheses: H p : The score s arises from the distribution of scores obtained by pairing suspect images in the suspectcontrol data set with the suspect images in the suspectreference data set. H d : The score s arises from the distribution of scores obtained by pairing suspect images in the suspectcontrol data set with the reference images in the potential population database. The difficulty in following the suspect-anchored approach is that in most cases it might not be possible to obtain a large set of data from the suspect in similar conditions to the trace. The lack of enough calibration data increases the uncertainty in the resultant estimate of a LR. Suspect-independent approach Certain specific solutions have appeared as how to increase the number of the within-source and the betweensource scores to estimate the distribution of scores under the prosecution and defense hypothesis [9, ]. A general solution to this problem is to compute the within-source scores by considering pairs of images from multiple potential suspects in the potential population database. Excluding images of the original suspect, images in the suspectcontrol and suspect-reference database of all the other suspects are paired with each other where each pair originate from the same source. This is referred to as suspectindependent approach. Similarly to obtain the suspectindependent between-source scores, reference images in the potential population are paired with the suspect-control data set of each suspect where each pair originate from different sources [8, ]. Using suspect-independent approach to

4 LR computation implies the following interpretations of the prosecution and defense hypotheses: H p : s arises from the distribution of scores obtained by pairing images in the suspect-control and suspect-reference data set of all the suspects in potential population where the paired images are obtained from the same source. H d : s arises from the distribution of scores obtained by pairing images in the suspect-control and suspect-reference data set of all the suspects in potential population where the paired images are obtained from different sources. For the between-source scores, besides the suspectanchored and suspect-independent approaches, another commonly used approach is to compute trace-anchored scores. In this approach, trace image is compared with all the reference images of the potential population to compute the between-source scores [].. Comparing the resultant LRs Since it is preferable to compute a suspect-anchored LR, there is some research on how to compute a LR for a biometric comparison when there is a limited calibration data available from the suspect. Ramos [] proposed a strategy which is based on the adaptation of the suspectindependent within-source score distribution to the suspectanchored scores via Maximum A Posteriori (MAP) estimation. Similarly, in forensic handwriting recognition, Davis [9] generated simulated writing samples from a small set of suspect samples to form a database for computation of the suspect-anchored within-source scores. These specific approaches do not generalize in most cases and usually a suspect-independent approach is considered as a last resort to compute a reliable LR for the evidence s []. In [] suspect-independent approach is proposed as a feasible alternative when a single sample is available from the suspect. Given the common use of the suspect-independent approach as an alternative to the suspect-anchored approach, it is important to study and analyse the differences in the score-to-lr functions produced by these approaches. Quantifying the variability between the suspectanchored and suspect-independent approach is still under investigation in most of biometric modalities including face recognition. Ramos [] studied the effect of using suspect-independent within-source scores instead of suspect-anchored approach on the resultant LRs in the context of speaker recognition. [9] describes the effect(s) of different calibrations data used to construct the denominator distribution in the context of handwriting recognition. We compute two functions from score to LLR using the suspect-anchored and suspect-independent sets of calibration scores. The behavior of these two score-to-llr functions is studied in different regions of the score-axis which correponds to different evidence values. For a more quantitative evaluation, we generate evidence values by uniformly sampling the score-axis and compute the number of cases in which these two approaches agree and disagree on a given range of LRs. A disagreement is reported when one approach produces a LR that falls into a different range. These ranges correspond to verbal equivalents which can be used in certain situations to report the forensic evaluation of the evidence. These ranges along with their corresponding verbal equivalnets are shown in table []. 5. Experimental setup To simulate a forensic case, we randomly select a small subset of five subjects from the FRGC [] database. Each subject has 6 frontal images taken in different illumination condition. For each subject, half of the images are used to create the suspect-control database while the remaining half are used as a suspect-reference database. Images in the suspect-control database are degraded by adding motion blur of 5 pixels with zero angle and downsampling them by half of the original resolution. Face regions are manually cropped where eye detection is performed automatically by the face recognition system []. Figure shows an example of the degradation applied to an image for creation of the suspect-control data set. The goal of the degradation process is to make the images in the suspect-control data set similar to a trace image. Insert motion blur Downsample Figure : An example of the degradation process applied to obtain suspect-control data set. Figure illustrates computation of the within-source and the between-source sets in each approach for a single image per subject assuming subject is the suspect. Table shows the number of unique comparisons (and hence the number of scores) in each appraoch of the withinsource and the between-source scores sets computation given 5 subjects and 8 images per subject in the probe and gallery set.

5 subject subject subject 5 subject subject subject 5 subject subject subject 5 S(,) Suspect-anchored within-source score S(,).. Suspect-independent within-source scores S(5,5) subject subject subject 5 S(,).. S(,5) S(,) S(,) S(,5) (a) S(5,) S(5,) S(,) Suspect-anchored between-source scores Suspect-independent between-source scores (b) Figure : The within-source and the between-source scores sets assuming the first subject as the suspect and image per subject. a) Computation of the within-source scores sets b) Computation of the between-source scores sets. within-source scores Suspect-anchored 8 X 8 = Suspect-independent X = 96 between-source scores Suspect-anchored 8 X (8 X ) = 96 Suspect-independent X 96 = 58 Table : Number of scores in the set of the within-source and the between-source scores. 6. Experimental results The score-axis is mapped to LLR using suspectanchored and suspect-independent approach in order to compare their score-to-llr mapping functions. Figure shows the frequency histograms of the scores in the within-source and the between-source sets, the ROCs of the calibration scores in the suspect-anchored and suspectindependent approach and the score-to-llr functions obtained by the ROCCH procedure as described in section.. Note the greater variation in the within-source scores for the suspect-anchored approach. Images of each subject are selected in such a way so that the variation in illumination and facial expressions are as close as possible across different subjects. Other conditions such as resolution and face pose is the same across all the subjects. However, still we observe considerable variation in the shape of the suspectanchored frequency histograms of the within-source scores. Variation in the histograms of the suspect-anchored withinsource scores are caused by either the slight difference in the illumination conditions and facial expressions or due to the difference in identity. Illumination conditions and facial expressions are very similar across different subjects, however, they are not the same for all 6 images of each subject. Besides the slight difference in illumination condition and facial expression, identity itself has effect on the suspect-anchored within-source scores distribution. A face recognition algorithm may perform differently for different subjects when it is used to match images of the same subject. Generally, it is expected that the suspect-anchored approach produces scores and subsequent LRs which are more discriminative as in the case of the first three subjects. However, this is not true in case of subject and 5 where the suspect-anchored within-source scores have more standard deviation than the suspect-independent within-source scores. There is a significant variation in the values of the LRs computed using the suspect-anchored and suspectindependent approach. For example in the case of subject, at the score location of., the suspect-anchored and the suspect-independent LR is 5 and 78 respectively. The horizontal lines in the mapping functions are due to the proposed strategy to avoid infinite LLRs. Values of LRs along this horizontal line are referred to as saturated LRs. These LRs, as illustrated later, depend on the size of the calibration scores used to map score-axis to LRs. In this region of LRs, the suspect-independent approach results in higher values of LRs than the suspect-anchored approach. Given the fact that LRs in the higher ranges are more important and useful in practice than in the lower and in the middle ranges, it can be argued that, in practice, the anchoring plays a crucial role. Note that the suspect-independent mapping functions across different subjects can be considered to reflect the possible variation in s when a different suspect is considered keeping the calibration data constant. In most cases the exact numerical value of a likelihood ratio is of less importance than the range in which it lies. These ranges can be taken into consideration when performing such a comparative study. The score-axis is uniformly sampled to simulate values of evidence s. These scores are converted to LLRs using each of the LR computation approach. Table shows the number of cases in which the two approaches compute LLRs which fall into the same range. As seen from table, in 96 cases out of 5, the two LRs agree on same verbal equivalents resulting in 59.% agreement rate. A considerable difference results from the fact that for subject and subject 5, the saturated LRs are in different ranges. Note that using a different face recognition system to compute scores and using a different method to map scoreaxis to LRs might lead to completely different results. Similarly a different database of images might also influence the variations between the suspect-anchored and suspect-

6 Within source Betweem source Suspect anchored Suspect independent Log Log Log Log Log Figure : The first two columns show the frequency histograms of the suspect-anchored () and suspect-independent () within-source and between-source scores sets. The third columns plots the ROCs from the corresponding sets of the withinsource and between-source scores. Last column shows the mapping function from score to LLR using the ROCCH procedure. Row through 5 repeat the same experiment considering each of the 5 subjects in the selected subset as the suspect.

7 Ranges Verbal equivalents Number of agreements P P P P P5 Total < LLR Very strong evidence to support H p < LLR Strong evidence to support H p < LLR Moderately strong evidence to support H p < LLR Moderate evidence to support H p < LLR Limited evidence to support H p < LLR Limited evidence to support H d 6 5 -< LLR - Moderate evidence to support H d 5 7 -< LLR - Moderately strong evidence to support H d 8 -< LLR - Strong evidence to support H d LLR < - Very strong evidence to support H d 5 Total Table : Number of times in which the LRs computed by the two approaches falls into same ranges. For each subject considered as the suspect, there are values of s generated by uniformly sampling the score-axis. Out of a total of 5 LRs computed by the two approaches, 96 times the LRs agree on one range of LLRs. Suspect anchored Suspect independent Log Log Log Log Log Figure 5: -axis is mapped to LLRs using the same sizes of the within-source and the between-source sets in the suspectanchored and suspect-independent approach. independent approach of LR computation. The effect of the difference in the size of the calibration sets between the suspect-anchored and suspect-independent approach can be investigated by randonly sampling without replacement a number of scores equal to the size of the suspect-anchored sets from the suspect-independent sets. Given the size of the within-source and between-source sets in the suspect-anchored and suspect-independent approach is the same, the variation in LRs is only caused by the nature of the distributions of the scores. Figure 5 shows the mapping function obtained by the two approaches when the within-source and the between-source sets are equally sized by random subsampling the suspect-independent withinsource and between-source sets so that the sizes of these sets in the suspect-independent approach is equal to those in the suspect-anchored approach. Note that reduction in the size of the calibration scores reduces the range of LRs that can be computed. This can be seen by comparing the mapping functions of the suspect-independent approach in figure and figure 5. Besides the saturated region of LRs, the difference in the size of the calibration sets has less effect on the resultant mapping function from score to LLRs. 7. Conclusions and future work We discussed the effect of different calibration data on the resultant forensic LR in the context of face recognition. The process of conversion of a score, obtained from the comparison of the crime scene image with the suspect image, to a forensic LR is described. It is observed that there is a significant variation between the LRs obtained using suspect-anchored and suspect-independent approach. The differences are more prominent in the higher ranges of LRs and therefore more caution should be taken if one approach is used as an altenative to the other. Future work will include quantifying the influence of images from other databases, different face recognition systems and other score-to-lr conversion methods. Furthermore, it is also of interest to study and compare these results from other modalities such as speech.

8 8. Acknowledgement The research is funded by the European commission as Marie-Curie ITN-project (FP7-PEOPLE-ITN-8) Bayesian Biometrics for Forensics. We would like to thank Cognitec Systems GmbH. for supporting our research by providing the FaceVACS software. Results obtained for FaceVACS were produced in experiments conducted by the University of Twente, and should therefore not be construed as a vendor s maximum effort full capability result. References [] A. K. Jain, P. Flynn and A. Ross. Handbook of Biometrics. Springer-Verlag: 7 [] Lucy D. Introduction to statistics for forensic scientists. West Sussex. John Wiley Sons, Inc: 5. [] C.G.G. Aitken, F. Taroni. Statistics and the evaluation of forensic evidence for forensic scientist. nd ed, Wiley, Chichester, UK:. [] B. Robertson, G.A. Vignaux. Interpreting evidence. Wiley, Chichester, UK: 995. [5] G.S. Morrison. Forensic voice comparison. in: I. Freckelton, H. Selby (Eds.), Expert Evidence, Thomson Reuters, Sydney, Australia: ch 99,. [6] C. Champod, D. Meuwly. The inference of identity in forensic speaker recognition. Speech Communication: 9-,, doi:.6/s67-69(99)78-. [7] P. Rose. Technical forensic speaker recognition. Computer Speech and Language : 59-9, 6 doi:.6/j.csl [8] Ramos Castro D. Forensic evaluation of the evidence using automatic speaker recognition systems [PhD dissertation]. Madrid (Spain): Universidad Autonoma de Madrid, 7. [9] C. Neumann, I.W. Evett, J. Skerrett. Quantifying the weight of evidence from a forensic fingerprint comparison: a new paradigm. Journal of the Royal Statistical Society: Series A 75 (), -6,. [] C. Peacock, A. Goode and A. Brett. Automatic forensic face recognition from digital images. Sci. Justice (): 9-,. [] T. Ali, L.J. Spreeuwers, and R.N.J. Veldhuis. Towards automatic forensic face recognition. In: International Conference on Informatics Engineering and Information Science (ICIEIS), Kuala Lumpur, Malaysia, Communications in Computer and Information Science 5: 7-55, Springer Verlag, ISSN ,. [] A.B. Hepler, C.P. Saunders, L.J. Davis, J. Buscaglia. -based s for handwriting evidence. Forensic Science International, 9: 9. doi:.6/j.forsciint...9,. [] Cognitec FaceVACS SDK version 8..:. [] G. Doddington, W. Liggett, A. Martin, M. Przybocki, and D. Reynolds, Sheep, Goats, Lambs and Wolves: A Statistical Analysis of Speaker Performance in the NIST 998 Speaker Recognition Evaluation, Proc. Intl Conf. Spoken Language Processing, 998. [5] T. Ali. and L.J. Spreeuwers and R.N.J. Veldhuis. A review of calibration methods for biometric systems in forensic applications. In: rd WIC Symposium on Information Theory in the Benelux, Boekelo, Netherlands: 6-,ISBN ,. [6] T. Fawcett and A. Niculescu-Mizil. PAV and the ROC convex hull. Machine Learning, 68(): 97-6, 7. [7] N. Brummer, J. Preez. Application-independent evaluation of speaker detection. Comput. Speech Lang. : 75, 6. [8] A. Nordgaard, T. Hoglund. Assessment of approximate likelihood ratios from continuous distributions: a case study of digital camera identification, Journal of Forensic Sciences 56: 9,. [9] L.J. Davis, C.P. Saunders, A.B. Hepler, J. Buscaglia. Using subsampling to estimate the strength of handwriting evidence via score-based s, Forensic Science International, 9: 9-,. [] D. Ramos,, J. Gonzalez-Rodriguez, G. Zadora, J. Zieba- Palus, C. Aitken. Information-theoretical comparison of likelihood-ratio methods of forensic evidence evaluation. In: Int. Symposium on Information Assurance and Security/ IWCF, Manchester, UK, IEEE-CS Press: 7 [] D. Meuwly, Forensic Individualization from Biometric Data. Science and Justice, 6(): 5-, 6. [] D. Ramos-Castro, J. Gonzlez-Rodrguez, A. Montero- Asenjo, J. Ortega-Garca. Suspect-adapted map estimation of within-source distributions in generative likelihood ratio estimation. Proceedings of the IEEE Odyssey Speaker and Language Recognition Workshop, doi:.9/odyssey.6.89.science International 6: 657,. [] F. Botti, A. Alexander, A. Drygajlo. An interpretation framework for the evaluation of evidence in forensic automatic speaker recognition with limited suspect data. In: Proceedings of Odyssey, the Speaker and Language Recognition Workshop, Toledo, Spain: 668,. [] P. J. Phillips, P.J. Flynn, T. Scruggs, K. Bowyer, J. Chang, K. Hoffman, J. Marques, J. Min, W. Worek. Overview of the Face Recognition Grand Challenge. CVPR: 5.