Journal of Electronc Imagng 14(2), 023003 (Apr Jun 2005) Perceptual mage qualty: Effects of tone characterstcs Peter B. Delahunt Exponent Inc. 149 Commonwealth Drve Menlo Park, Calforna 94025 Xueme Zhang Aglent Technologes Laboratores 3500 Deer Creek Road, MS 26M-3 Palo Alto, Calforna 94304 Davd H. Branard Unversty of Pennsylvana Department of Psychology 3401 Walnut Street, Sute 302C Phladelpha, Pennsylvana 19104 Abstract. Tone mappng refers to the converson of lumnance values recorded by a dgtal camera or other acquston devce, to the lumnance levels avalable from an output devce, such as a montor or a prnter. Tone mappng can mprove the appearance of rendered mages. Although there are a varety of algorthms avalable, there s lttle nformaton about the mage tone characterstcs that produce pleasng mages. We devsed an experment where preferences for mages wth dfferent tone characterstcs were measured. The results ndcate that there s a systematc relaton between mage tone characterstcs and perceptual mage qualty for mages contanng faces. For these mages, a mean face lumnance level of 46 49 CIELAB L* unts and a lumnance standard devaton (taken over the whole mage) of 18 CIELAB L* unts produced the best renderngs. Ths nformaton s relevant for the desgn of tone-mappng algorthms, partcularly as many mages taken by dgtal camera users nclude faces. 2005 SPIE and IS&T. [DOI: 10.1117/1.1900134] 1 Introducton Consumers of dgtal cameras and related products desre hgh-qualty mages. Consumer preference for mages, however, s not easy to predct. Even f t were techncally feasble, creatng a perfect reproducton of the lght that arrved at the camera would not guarantee the most preferred renderng of the orgnal scene. For example most professonal portrature employs a large degree of mage enhancement, and the results are almost always preferred to a verdcal renderng. Ths may occur because most consumers judge the attractveness of an mage wthout drect reference to the orgnal scene, so that ther judgments are based on memory, ether of the specfc scene or of generc scenes. There s evdence that memory for colored objects can be unrelable. 1 3 Paper 03007 receved Jan. 21, 2003; revsed manuscrpt receved Aug. 7, 2003; accepted for publcaton Aug. 17, 2004; publshed onlne May 12, 2005. 1017-9909/2005/$22.00 2005 SPIE and IS&T. Dgtal mages may be modfed through the applcaton of mage processng algorthms, but what modfcatons make mages look better s not well understood. One approach to ths problem s to study drectly the effect of mage processng on mage preference. We recently examned the perceptual performance of demosacng algorthms n ths manner. 4 Prevous work has also studed the relaton between mage colorfulness and human observer qualty/ naturalness ratngs. 5 7 Here we apply smlar expermental methods to study the relaton between mage tone characterstcs and perceptual mage qualty. Tone mappng refers to the converson of nput lumnance values, as captured by an acquston devce e.g., a dgtal camera, to lumnance values for dsplay on an output devce e.g., a computer montor. Lumnance values n a natural mage can range over about fve orders of magntude. 8 Ths compares to a much smaller range of about two orders of magntude avalable wth a computer montor under typcal vewng condtons. Even for the usual stuaton where the mage acquston devce quantzes the number of lumnance levels to match the number of levels avalable on the output devce, tone mappng can stll mprove the appearance of an mage. The relaton between nput and output lumnance values produced by a tone-mappng algorthm s called a tone-mappng curve. Tone mappng changes the tone characterstcs of the mage. By tone characterstcs we mean the dstrbuton of the lumnance values of the mage s pxels, wthout regard to how the pxels are arranged spatally. In general, tone characterstcs can ether be assessed globally over the entre mage, or locally over some smaller regon of nterest. Wthn an mage regon ether global or local, tone characterstcs are completely descrbed by the lumnance hstogram of the regon. Ths specfes the number of mage Journal of Electronc Imagng 023003-1
Delahunt, Zhang, and Branard: Perceptual mage qualty... pxels wthn the regon that have each possble output lumnance value. In ths paper, we wll consder both global and local tone characterstcs. Prevous work on tone mappng has focused on comparsons of the performance of dfferent tone-mappng methods. Much of ths work was conducted n the context of flm-based photography, where practcal consderatons lmted attenton to global tone-mappng methods n whch a sngle tone-mappng curve was appled to the entre mage see revew by Nelson 9. Bartleson and Breneman 10 suggested that a good tone-mappng curve establshed a 1:1 relaton between relatve perceved brghtness values n the scene and the rendered mage, where relatve brghtnesses were computed usng a modfed power functon derved from research on brghtness scalng. 11 Ther curve corresponded closely to curves that receved hgh ratngs n a psychophyscal study performed by Clark. 12 Further work by Hunt and co-workers 13,14 suggested that the Bartleson and Breneman prncple 10 should be modfed dependng on the vewng condtons n partcular the surround of the mage and suggested that although a lnear relaton between scene and mage relatve brghtnesses was approprate for reflecton prnts, a power-law relaton between relatve brghtnesses was more approprate for transparences. The wdely used zone system for photographc tone mappng revewed n Renhard et al. 15 reles on perceptual judgments of how regons n the orgnal scene appeared to the photographer. In flm photography, t s not practcal to automatcally adjust the tone-mappng curve between mages at separate locatons wthn an mage, snce the shape of these curves s governed by physcal characterstcs of the emulsons and flm-development process. Wth the advent of dgtal magng, a wder range of tone-mappng algorthms become of practcal nterest. On the other hand, n many dgtal cameras mage quantzaton precedes the applcaton of a tonemappng algorthm, a feature that ncreases the challenges for successful tone mappng. Thus there has been renewed nterest n developng tone-mappng algorthms see, e.g., Refs. 8 and 16 18. Evaluaton of these methods has agan emphaszed comparng the output of competng algorthms. A recent study by Drago et al., 19 for example, appled seven tone-mappng technques to four dgtal mages and ther performance was rank ordered based on observer preferences. Algorthms that apply a fxed tone-mappng curve to any mage have the feature that the tone characterstcs of the mages produced by the algorthm can vary wdely, snce these characterstcs depend strongly on the nput. Dgtal magng presents the opportunty to develop algorthms usng a dfferent prncple. Rather than defnng the relatonshp between nput and output lumnances, one can specfy target output tone characterstcs and apply an magedependent transformaton that yelds a good approxmaton of these characterstcs. One early dgtal tone-mappng algorthm, hstogram equalzaton, s based on ths dea: the algorthm maps the lumnance values n the nput mage to produce a desred lumnance hstogram n the output mage. Although t seems unlkely that the optmal output hstogram s completely ndependent of mage content, the prncple of specfyng target output mage tone characterstcs has been ncorporated nto recent tone-mappng algorthms ntended to mprove upon hstogram equalzaton. In these algorthms, the output hstogram vares wth an analyss of mage content. 8,16 The work we present here s ntended to further explore the dea that effectve tone mappng can be acheved through specfcaton of desred output mage tone characterstcs. Rather than focusng on the development and evaluaton of tone-mappng algorthms, we chose to address the underlyng ssue of whether we could dentfy output tone characterstcs that produce perceptually attractve mages, and whether such characterstcs depend on mage content. To ths end, we report the results of two mage preference studes and analyze how mage preference s related to mage tone characterstcs. The work presented here employs mages captured wth standard dgtal cameras and s drected at mprovng the qualty of mages produced from such cameras. We do not explctly consder the case where the dynamc range of the capture and dsplay devces vares greatly see Refs. 8, 17, 18, and 20. As most amateur dgtal photographs nclude people, our studes employ an mage set that conssted manly of mages of people. We also wanted to nclude mages of people from dfferent ethnc backgrounds, snce many earler tonemappng studes used mages of Caucasans only e.g. Refs. 12, 21, and 22. 2 Experment 1 2.1 Overvew Experment 1 was exploratory, wth the goal of dentfyng systematc relatonshps between tone varables and mage qualty. We appled four dfferent tone-mappng methods to each of 25 expermental mages and measured the perceptual qualty of the dfferent renderngs of each mage. These algorthms produced output mages wth a range of tone characterstcs. Image preference was measured usng a parwse comparson procedure. On each tral, observers ndcated whch of two presented mages was the most attractve. The parwse comparson procedure s ntutve for observers and yelds relable data. 4 Note, however, that observers only make judgments about dfferent renderngs of the same nput mage. Thus some analyss s requred to aggregate a data set large enough to explore the queston of how an mage s tone characterstcs relate to ts perceptual qualty. To ths end, the preference choce data were analyzed usng a regresson procedure 23 to yeld metrc dfferences n mage qualty between mage pars. The procedure yelds dfference ratngs that are commensurate across nput mages. We then asked whether dfferences n specfc mage tone varables were predctve of the dfference ratngs. Here the term tone varable refers to a summary measure, such as mean lumnance, that may be computed from the output lumnance hstogram. We used 25 dgtally acqured mages and rendered each on a CRT computer montor usng four dfferent tonemappng methods. The four methods produced results that were perceptually dfferent for most of the mages, thus provdng varaton n mage tone characterstcs whose effect we could study. Journal of Electronc Imagng 023003-2
Delahunt, Zhang, and Branard: Perceptual mage qualty... 2.2 Methods: Image Acquston Twenty-fve mages were used n Experment 1. Twentyone were captured n Santa Barbara, Calforna and four were taken n Palo Alto, Calforna. All of the mages were captured under daylght, at dfferent tmes of the day, throughout May 1999. The llumnant was measured mmedately followng the acquston of each mage by placng a whte reflectance standard n the scene and measurng the reflected lght usng a Photo Research PR-650 spectraradometer. Of the 25 mages, 17 were portrats of people, 5 were landscapes, and 3 were of objects. The 21 Santa Barbara mages were taken wth a Kodak DCS-200 camera and the 4 Palo Alto mages wth a Kodak DCS-420 camera. Both cameras have a resoluton of 1524 1012 wth RGB sensors arranged n a Bayer mosac. 24 The DCS-200 captures the nput lght ntensty usng 8-bt lnear quantzaton, whereas the DCS-420 captures wth 12- bt precson. The 12-bt values captured by the DCS-420 are converted to 8-bt values on-camera va a nonlnear transformaton. The relatve RGB spectral senstvtes and response propertes of both cameras were characterzed as descrbed elsewhere. 25 Ths characterzaton left one free parameter descrbng the overall senstvty of the camera undetermned, as ths parameter vares wth acquston exposure duraton and f-stop. The mages were cropped to a maxmum sze of 575 w by 800 h pxels to ensure that two renderngs of each mage could be dsplayed smultaneously on the computer screen used n our experment. 2.3 Image Processng 2.3.1 Dark level subtracton For the DCS-200, a dark level was subtracted from the raw quantzed pxel values before further processng. The dark level was estmated from an mage acqured wth the lens cap on and computng the spatal average of the resultng mage. The average for the red, green, and blue sensors were all 13.5 on the camera s 8-bt 0 255 output scale and ths s the value that was subtracted. To estmate the dynamc range of the mages, we compared the mnmum and maxmum pxel values for the green sensor. These typcally occuped the entre allowable output range approxmately 13 255 before dark subtracton. Gven that some pxels had values near zero after dark subtracton, t s not possble to express the dynamc range of these mages as a meanngful rato. For the DCS-420, t was possble to lnearze the output values usng a look-up table provded as part of each raw mage fle. Ths was done pror to further processng. After lnearzaton, the estmated dark level for the DCS-420 was close to zero and no explct dark level subtracton was performed. The dynamc range of these mages could be estmated by takng the rato of the maxmum to mnmum lnearzed output value for the green sensor. These ratos vared from 17 to 140 across the DCS-420 mages used n ths experment. 2.3.2 Demosacng Because the two cameras employed a mosaced desgn, wth each raw pxel correspondng to only one of the three sensor types, t was necessary to apply a demosacng algorthm to convert the raw mosaced mage to a full color RGB mage. We used a Bayesan demosacng algorthm developed by Branard and colleague 26 28 and summarzed n a recent paper 4 where we evaluated the perceptual qualty of demosacng algorthms. The performance of the Bayesan algorthm s controlled by a number of parameters. For the applcaton here, the correlaton between nearest-neghbor pxels was assumed to be 0.90, whereas the correlaton between the responses of dfferent sensor classes at the same mage locaton was estmated from a blnear nterpolaton of the mosaced mage. Fnally, the algorthm assumed that there was addtve normally dstrbuted pxel nose wth a standard devaton for each sensor class equal to 4% of the spatal average of responses for that class. The estmates at each locaton were obtaned by applyng the algorthm to a 5 5 mage regon surroundng that pxel. The demosacng results for our mages were n general qute good, wth very few notceable artfacts. 2.3.3 Color balancng We by-passed the on-board color balancng of the cameras and used our measurements of the scene llumnants to color balance the mages. Gven the camera s RGB sensor relatve spectral senstvtes and the measured llumnant, we were able to estmate the relatve surface spectral reflectance of the object at each scene locaton. Ths was done usng a Bayesan estmaton procedure that wll be descrbed n a future report. Brefly, we constructed a normally dstrbuted multvarate pror dstrbuton for object surface reflectances by analyzng the Vrehl et al. 29 data set of measured surface reflectance functons. The analyss followed closely the method ntroduced by Branard and Freeman 30 n ther work on computatonal color constancy. Gven the pror, estmatng reflectances from the sensor responses s a straghtforward applcaton of Bayes rule. Usng the estmated surface reflectance functons, we could then synthesze an mage that conssted of the CIE XYZ trstmulus coordnates that would have been obtaned had the surface been vewed under standard CIE daylght D65, up to an overall scale factor. Ths scale factor vared from mage to mage dependng on the scene llumnant, acquston exposure, and acquston f-stop. Uncertanty about the scale factor s equvalent to uncertanty about the overall ntensty of the scene llumnant and s thus handled transparently by the tone-mappng algorthms that we appled to render the mages, whch are desgned to apply to mages captured over a wde range of overall scene lumnances. Note that mage L* propertes reported n ths paper refer to L* values for the expermental mages dsplayed on the expermental montor, not to L* propertes of regons of the orgnal scene. To check the accuracy of the color balancng process, an mage of a Macbeth color checker was taken usng the Kodak DCS-420 dgtal camera. Raw RGB values before demosacng were extracted for each of the 24 color checker patches. The Bayes color correcton was used to estmate the XYZ values of the patches under CIE D65 llumnaton. These estmates were compared wth target values computed from measured spectral reflectances of the color checker patches and the known spectral relatve spectral power dstrbuton of CIE daylght D65. Here the free overall scale factor was determned so that the two mddle gray color checker patches patch Nos. 21 and 22 matched Journal of Electronc Imagng 023003-3
Delahunt, Zhang, and Branard: Perceptual mage qualty... Fg. 1 Tone mappng. The top panel n the fgure shows the global L* lumnance hstogram of the orgnal mage. The four panels n the frst full row show the hstograms after applcaton of the four tone-mappng algorthms. The four panels n the mddle row show the tone-mappng curves used by the four algorthms for the mage shown. The bottom panels show the output mages for each of the four algorthms. n average lumnance between the color balanced and target values. The average CIELAB E 94 dfference between the estmated values and drectly determned target values average taken over the 24 patches was 3.6 unts, ndcatng that the algorthm worked well. 2.3.4 Tone mappng Four tone-mappng algorthms Clppng, Hstogram Equalzaton, Larson s Method, and Holm s Method were appled to the color balanced XYZ mages. These are descrbed below. Each method transformed the lumnance of each mage pxel whle holdng the chromatcty of each pxel constant. The relaton between a partcular measurement of nput and output lumnance s referred to as the algorthm s tone-mappng curve. In general, the tonemappng curve produced by an algorthm depends on mage content. Each of the algorthms used was global, n the sense that the same tone-mappng curve was appled to every pxel n the mage. It should be emphaszed that our man goal was to use a varety of tone-mappng algorthms that would produce dfferent tone-mappng characterstcs. The performance of each algorthm was not of prmary concern. All four methods led to acceptable as judged by the authors renderngs for all of the mages. In Fg. 1 we show an example of the hstograms, tone-mappng curves, and output mages produced by the four algorthms. 1 Clppng: For the clppng method, the tonemappng curve relatng mage lumnance to dsplay lumnance was a straght lne through the orgn. Image lum- Journal of Electronc Imagng 023003-4
Delahunt, Zhang, and Branard: Perceptual mage qualty... nances that were mapped to dsplay lumnances greater than the maxmum avalable on the output devce were clpped to the maxmum. The slope of the tone-mappng curve was determned so that maxmum dsplay lumnance was equal to fve tmes the mean lumnance of the tonemapped mage. Ths clppng method provdes a smple baselne that works reasonably well. 2 Hstogram equalzaton: A wdely used method that re-assgns lumnance values to acheve a partcular target lumnance hstogram e.g., unform or Gaussan n the tone-mapped mage. 31 Ths method effcently uses the dynamc range of the dsplay devce, but can generate mages that have exaggerated contrast and thus a harsh appearance. In our mplementaton, the target hstogram was a Gaussan centered at the mddle of the output range. 3 Larson method: A more sophstcated verson of hstogram equalzaton. The dea s to lmt the magntude of lumnance mappng, so that lumnance dfferences wthn the mage that were not vsble before tone mappng are not made vsble by t. Images tone mapped wth the Larson method generally have a more natural appearance than when usng the tradtonal hstogram equalzaton method. 4 Holm s method Ref. 16 : Part of a color reproducton ppelne created at Hewlett-Packard Labs for use n dgtal cameras. We used only the tone-mappng segment of the ppelne for consstency wth the other methods. In Holm s method, the nput mage s frst classfed as one of several dfferent types e.g., hgh key or low key usng a set of mage statstcs. A tone-mappng curve s then generated accordng to the mage type and mage statstcs, and ths curve s appled to the whole mage. Ths method ncorporates preference gudelnes that came from the nventor s extensve experence n photographc magng. 2.3.5 Renderng for dsplay The mages were presented on a CRT montor. Converson between the tone-mapped XYZ values and montor settngs was acheved usng the general model of montor performance and calbraton procedures descrbed by Branard. 32 The calbraton was performed usng the PR-650 spectraradometer. Spectral measurements were made at 4 nm ncrements between 380 and 780 nm but nterpolated wth a cubc splne to the CIE recommended wavelength samplng of 5 nm ncrements between 380 and 780 nm. CIE XYZ coordnates were computed wth respect to the CIE 1931 color matchng functons. 2.3.6 Room and dsplay setup The expermental room was set up accordng to the Internatonal Organzaton for Standardzaton Recommendatons for Vewng Condtons for Graphc Technology and Photography. 33 The walls were made of a medum gray materal and the table on whch the montor was placed was covered wth black cloth. The room was lt by two fluorescent celng lghts 3500 K controlled by a dmmer swtch set at a dm level. The llumnaton measured at the observer poston was 41 lux. The experment was controlled by MATLAB software based on the Psychophyscs Toolbox. 34,35 The mages were dsplayed on a Hewlett Packard P1100 21 n. montor 1280 1024 pxels drven by a Hewlett Packard Kayak XU computer. 2.3.7 Procedure On each tral of the experment, observers were shown pars of the same scene rendered va dfferent tone-mappng methods and were asked to choose the mage that they found to be the most attractve. To further explan ths nstructon, observers were asked to choose the mage they would select to put nto ther own photo album. The observers were also asked to look around the mages before makng a decson rather than focus on just one aspect. The experment started after a 2 mn adaptaton perod. Three seconds after each par of mages was presented, two selecton boxes appeared under the mages. Ths 3 s delay was to encourage the observers to carefully consder ther decson. There was no upper lmt on response tme. The observers ndcated ther preference by usng a mouse to move a cursor to the selecton box under the preferred mage and clckng. The observer could subsequently change hs/her mnd by clckng on the alternatve box. When the observer was satsfed wth hs/her selecton, he/she clcked on an enter button to move to the next tral. The mages were vewed from a dstance of 60 cm. The mages ranged n wdth from 17 to 19 cm subtendng vsual angles 16.1 to 18.0 and ranged n heght from 13 to 25 cm subtendng vsual angles from 12.4 to 23.5. Images were shown n pars on the montor, one on the left and one on the rght. Each mage had a border of wdth 1 cm whch was rendered as the brghtest smulated D65 llumnant the montor could produce 78 cd/m 2. The remanng area of the montor emtted smulated D65 llumnant but at a lumnance level of about 20% of the border regon measured at 14.9 cd/m 2. Usng four renderng methods gves sx parwse presentaton combnatons per mage. For the 25 expermental mages, ths produces a stmulus set of 150 mage pars. 2.3.8 Observers Twenty observers partcpated n the experment 12 males and 8 females wth an average age of 31 range 19 62. The experment took place at Hewlett Packard Labs n Palo Alto and the observers were recruted by postng flyers around the buldng complex. The observers were a mxture of Hewlett Packard employees and outsde frends and famly. Only color normal observers partcpated. Color vson was tested usng the Ishhara color plates. 36 2.3.9 Data analyss The am of our analyss was to summarze mage tone characterstcs usng smple tone varables, and to determne whether these varables predcted mage preference. We hoped to dentfy systematc relatonshps between preference ratngs and tone varables. Thus our data analyss has two mportant components: the procedure used to transform the parwse mage judgments to mage preference ratngs and the procedures used to extract varables that capture mage tone-mappng characterstcs. 2.4 Image Ratngs The raw data conssted of parwse rankngs between the four dfferent renderngs of each mage. For each mage, we used a regresson based scalng method 23 to convert the parwse rankngs to preference ratngs for each of the four Journal of Electronc Imagng 023003-5
Delahunt, Zhang, and Branard: Perceptual mage qualty... versons. Denote these ratngs as j where the superscrpt denotes the mage (1 25) and the subscrpt j denotes the renderng verson (1 j 4, algorthms as numbered above. Wthn mage, these ratngs for the four dfferent versons of an mage are drectly comparable. Snce no preference judgments were made across mages, however, the ratngs across mages are not necessarly commensurate. Although we cannot make comparsons of preference ratngs across mages, we can make such comparsons of dfferences n preference ratngs. Under assumptons that we found reasonable, 23 the four ratngs generated for each mage le on an nterval scale. The unt of ths scale corresponds to one standard devaton of Gaussan perceptual nose that observers are assumed to experence when makng preference judgments, and the unt s thus common to the ratngs generated for all 25 mages. What dffers across mages s the orgn of the scale, whch s assgned arbtrarly by the regresson method. To remove the effect of orgn, we can compute dfference ratngs between the j th and k th renderngs, jk j k (1 j,k 4). Because the ratng scale constructed for each mage has a common unt, the dfference ratngs are commensurate across mages. Thus we can explore whether there are mage tone characterstcs whose dfferences predct dfference ratngs. From the four renderngs for each mage, we can take sx parwse dfferences. Only three of these are ndependent, however, n the sense that gven any three parwse dfferences the other three may be reconstructed. To avod ths redundancy, we used only the dfference ratngs 12, 23, and 34 1 25 n the analyss. 2.5 Image Tone Characterstcs To descrbe mage tone characterstcs, we used the L* coordnate of the CIELAB unform color space. 37 Ths measure of lumnance s normalzed to a whte pont, and the normalzed values are transformed so that equal dfferences n L* represent approxmately equal dfferences n the percepton of brghtness. The maxmum montor output all three phosphors set at the maxmum was used as the whte pont for convertng mage lumnance to L*. We consdered two summary measures of the L* hstogram: the mean L* value and the standard devaton of the L* values. For each mage, we denote the mean L* of the j th renderng value by j and the standard devaton of the L* values by j. These are both global tone varables, computed from the entre mage. Note that j s n essence a measure of the overall lumnance of the mage, whereas j s n essence a measure of mage contrast. A prelmnary analyss ndcated that to the extent mage qualty ratngs depended on the tone characterstcs j and j, ths dependence was not monotonc. Ths observaton makes ntutve sense. Consder the mean L* value j.an mage wth a j value equal to zero wll be entrely black and not provde a satsfactory renderng. Smlarly, an mage wth a very large j value wll be entrely whte. Clearly a renderng wth a j value between the two extremes s ndcated. Smlar arguments apply to j. Fg. 2 Face submages were created by croppng the faces out of the mages as llustrated. To account for a possble nonmonotoncty of the relaton between mage qualty and the tone characterstcs j and j, we consdered transforms of these varables: j j 0, j j 0. Here the parameter 0 represents the optmal value for j, that s the value that leads to the hghest mage qualty across all mages and renderngs, and thus devatons of j from 0 should correlate wth reduced mage qualty. Smlarly, the parameter 0 s the optmal value for j. 2.6 Analyss As noted prevously, our data set does not provde us wth drect access to mage qualty, but rather to qualty dfference ratngs jk between pars of mages. To ask whether mage tone characterstcs predct mage qualty, we nvestgated whether dfferences between the tone varables j and j predct the dfference ratngs jk. Specfcally, we defned the tone varables dfferences jk j k and jk j k and examned the lnear dependence of jk on each of these dfferences. Snce each transformed varable depends on ts correspondng optmal value, numercal parameter search over the optmal value was used to maxmze the predctve value (R 2 )of jk and jk. 2.7 Face Images In follow-up questonng conducted at the end of the experment, many observers commented that for mages contanng people, the appearance of faces was an mportant factor n ther decson makng. For mages contanng faces 17 of 25 we examned the face regons n more detal and defned face submages so the tone characterstcs of these regons could be extracted. The submages were defned by hand: an example of how a face submage was defned s shown n Fg. 2. One mage had two faces; only the foreground face was used for ths analyss. The faces were of varous ethnctes 8 Caucasan, 4 Afrcan- Amercan, 3 Asan, 1 Hspanc, and 1 Polynesan. For the mages contanng faces, we repeated our analyss of dfference ratngs when the tone characterstcs depended only on the pxels n the face submage. We denote 1 Journal of Electronc Imagng 023003-6
Delahunt, Zhang, and Branard: Perceptual mage qualty... Fg. 3 Predcton of dfference ratngs from global tone characterstcs. The fgure plots the dfference ratngs obtaned for all mages n Experment 1 aganst ũ jk (top panel) and jk (bottom panel). these dfference ratngs by face jk and face jk. Note that these are local tone varables, n that they depend only on a subregon of the entre mage. 3 Results Fgure 3 shows the dfference ratngs jk plotted aganst tone characterstc dfferences jk top panel and jk bottom panel for our entre data set. From the fgure, we can see that any systematc dependence of dfference ratngs on jk s weak at best, but that there s a clear dependence of the dfference ratngs on jk. Note that the negatve slope of the dependence shown n the bottom panel of Fg. 3 makes sense: f a renderng j s preferred to mage k postve dfference ratng jk ), then the devaton of mage j s L* standard devaton from ts optmal value s smaller than the correspondng devaton for mage k negatve jk ). These conclusons are confrmed by statstcal tests on the sgnfcance of the lnear relaton between the jk and each ndependent varable. The R 2 value for jk s small 0.07 but sgnfcant (p 0.05), whereas jk explans a substantal fracton of the varance (R 2 0.31, p 0.001). The optmal value found for 0 was 46.6, whereas that found for 0 was 17.8. The predctve value of jk and jk s greater for mages contanng faces than for nonface mages. The four panels of Fg. 4 show the dfference ratngs plotted aganst the jk top panels and jk bottom panels for the face left panels and nonface rght panels mages separately. The lnear predctve value of jk and jk s sgnfcant only for the face mages, and agan only the global L* standard devaton accounts for a substantal proporton of varance. Face mages, jk : R 2 0.09, p 0.05; face mages, jk : R 2 0.58, p 0.001; nonface mages, R 2 0.04, jk : p 0.34; nonface mages, jk : R 2 0.00, p 0.83.) We focused on the face mages for further analyss and consdered whether the local tone varable dfferences face jk and face jk extracted from the face regon provded addtonal predctve value. Fgure 5 plots the dfference ratngs for the face mages aganst these two addtonal varables. Both local tone characterstcs are predctve of the dfference ratngs ( face jk : R 2 0.59, p 0.001; face jk : R 2 0.29, p 0.001). The analyss prevously presented shows that both our global and local face regon tone characterstcs were predctve of mage qualty: dfferences n each varable separately are sgnfcantly correlated wth the dfference ratngs. We used multple regresson to ask how well all four varables could jontly predct mage qualty. The overall R 2 when the dfference ratngs were regressed on ũ jk, jk, face jk, and face jk was 75%. Stepwse regresson showed that almost all of the explanatory power was carred by two of the four varables: jk and face jk. These two varables alone provded an R 2 of 0.72. Fgure 6 shows the measured dfference ratngs for the face mages plotted aganst the predctons based on jk and face jk. If the two varables were perfect predctors of mage qualty, the data would fall along the dagonal lne. Recall that the data analyss nvolves fndng the optmal values for the tone varables jk and face jk. Fgure 7 shows a plot of how the R 2 measure for the face mages vares wth the optmal values 0 and face 0. The optmal value 0 was 17.8, whereas that for face 0 was 48.7. To test f the optmal values vared across ethnctes, we dvded the mages nto two groups 8 Caucasan mages and 9 non-caucasan mages and then re-ran the analyss. The results for the two groups were very smlar for face mean and standard devaton L* values ( face 0 values were, Caucasan mages: 48.6, non-caucasan mages: 48.8, and face 0 values were, Caucasan mages: 19.2, non- Caucasan mages: 18.4 but dffered somewhat for global L* standard devaton ( 0 values were, Caucasan mages: 15.4, non-caucasan mages: 20.7. Although the performance of each of the four algorthms was not of prmary concern n ths paper, a summary of the preferences s shown n Table 1 for completeness. Note that Holm s method performed partcularly well overall. The data from Experment 1 support the followng conclusons: We were unable to fnd a tone varable that predcted perceptual mage qualty for nonface mages. For face mages, a number of tone varables were sgnfcantly correlated wth the dfference ratngs. Two varables accounted for the majorty of the varance n the data that Journal of Electronc Imagng 023003-7
Delahunt, Zhang, and Branard: Perceptual mage qualty... Fg. 4 Predcton of dfference ratngs from global tone characterstcs, face mages (left panels) and nonface mages (rght panels) shown separately. The dfference ratngs obtaned for all mages n Experment 1 aganst ũ jk (top panels) and jk (bottom panels) are plotted. we could explan. These were the dfference n L* standard devatons across the entre mage ( jk ) and the mean L* value dfference for the face submage ( face jk ). The data allowed dentfcaton of optmal values for each of these varables. 4 Experment 2 The results from Experment 1 suggest that for mages contanng a face, the standard devaton of mage lumnance values and the mean lumnance level of the face tself do a good job of predctng predctve mage qualty. In Experment 2, we explored the effect of face mean lumnance n more detal. We used a dverse set of face mages that ncluded people wth a wde range of skn tones and mages wth multple faces. 4.1 Methods The methods were the same as for Experment 1 except for the followng. 4.1.1 Image acquston Images were acqured usng the Kodak DCS-420 dgtal camera. Ffteen mages were selected, all of whch were portrats taken under daylght. Face subregons were agan dentfed by hand. Ten contaned only one subject 5 Caucasan, 3 Afrcan-Amercan, 2 Asan and fve contaned multple subjects 1 of Caucasans only, 2 wth Afrcan Amercans only, and 3 wth a mxture of ethnctes. For the mages contanng multple faces, the dentfed face subregons ncluded all faces. The dynamc range of the mages, computed as descrbed for Experment 1, vared between 37 and 245. 4.1.2 Image processng We wanted to generate rendered mages wth dfferent face lumnance levels wth mnmal changes to the L* standard devaton. Ths was done by applyng a smooth global tonemappng curve to the mages, wth the curve parameters chosen so that the output mages had the desred face regon mean L* and L* standard devaton tone characterstcs. Face submages were selected by hand and 5 versons of each mage were created wth dfferent mean face L* target values 42, 48, 52, 56, and 62 and wth the L* standard devaton value held fxed at approxmately 18.8. Fve dfferent renderngs per mage produced ten possble parwse presentatons for each of the ffteen mages. Dfference ratngs 12, 23, 34, 45 and correspondng df- ferences n tone varables were used n the analyss. Journal of Electronc Imagng 023003-8
Delahunt, Zhang, and Branard: Perceptual mage qualty... Fg. 6 Measured dfference ratngs plotted aganst dfference ratngs predcted as the best lnear combnaton of jk and ũ face jk for the face mages of Experment 1. If the predctons were perfect, the ponts would fall on the dagonal lne. The error bars show 1 standard error of measurement for the dfference ratngs, computed usng a resamplng method (Ref. 41). The raw preference data were resampled 50 tmes. For each resamplng, dfference ratngs were computed and the standard devaton of the resultng dfference ratngs was taken as the standard error. Fg. 5 Predcton of dfference ratngs from face-regon tone characterstcs, face mages only. The dfference ratngs obtaned for the face mages n Experment 1 aganst ũ face jk (top panel) and face jk (bottom panel) are plotted. 4.1.3 Observers Nneteen color normal observers partcpated n the experment 12 males and 7 females wth an average age of 35 range 23 62. Eght of the observers had prevously partcpated n Experment 1. 4.2 Results The data were analyzed n the same fashon as were the data for Experment 1 wth respect to the predctve power of the face jk varable. The top panel of Fg. 8 shows a scatter plot of the dfference ratngs aganst mean faceregon L* value dfferences. For mages wth multple faces, the mean face L* value was used. The regresson results showed that ths tone characterstc dfference was sgnfcantly correlated wth the dfference ratngs ( p 0.001) and that percent varance explaned was R 2 0.49. Ths replcates and extends the results of Experment 1 wth respect to ths tone characterstc. After the experment, observers were gven a chance to provde comments and feedback. In Experment 2, a number of observers noted that some renderngs of three of the mages contaned vsble artfacts n the facal regons, and that these artfacts had a strong negatve nfluence on ther preference for those mages. Post-hoc examnaton of the mages confrmed the observer reports. We beleve the artfacts arose because the tone-mappng procedure amplfed the nose n some of the darker mage regons. Because our nterest was n tone characterstcs, not artfacts, t seemed of nterest to repeat the analyss wth the three problematc mages excluded. Ths led to an ncrease n the percent of varance accounted for by the face L* mean dfference varable, wth R 2 0.66 rather than 0.49. The bottom panel of Fg. 8 shows the relaton between dfference ratngs and ths varable after the excluson. As part of the analyss, numercal search was agan used to fnd value face 0 that optmzed R 2. Ths value was 49.2 when the full data set was analyzed and 46.5 wth the three mages excluded, both very close to the value of 48.7 found n the frst experment. The dependence of the R 2 value on the optmal parameter s shown n Fg. 9 for the two cases. We examned f the optmal face 0 value vared across ethnctes. The mages were dvded nto two groups 4 mages of Caucasans, 6 mages of non-caucasans. Fve of the mages were not ncluded 2 had multple faces of dfferent ethnctes and three has vsble artfacts n the face regon as dscussed above. We re-ran the analyss and the results for the two groups were very smlar ( face 0 values were, Caucasan mages: 46.3, non-caucasan mages: 45.7. 5 Dscusson 5.1 Summary The paper presents experments that explore whether a number of smple mage tone characterstcs are predctve of perceptual mage qualty. For the nonface mages we studed, we were unable to dentfy any such varables. For mages consstng prmarly of faces, however, the results suggest that the best mage qualty results when the face L* Journal of Electronc Imagng 023003-9
Delahunt, Zhang, and Branard: Perceptual mage qualty... Fg. 7 Optmal values 0 and u face 0 for the face mages used n Experment 1. Each panel plots the percent varance explaned by a sngle tone characterstc (top panel: jk ; bottom panel: ũ face jk )as a functon of the correspondng optmal value (top panel: 0 ; bottom panel: u face 0 ). Fg. 8 Predcton of dfference ratngs from face-regon tone characterstcs, for Experment 2. The dfference ratngs aganst ũ face jk are plotted. The top panel shows the full data set and the bottom panels shows the data when three mages wth artfacts were excluded. lumnance s n the range 46 49, and the standard devaton of the mage L* lumnances s approxmately 18. Ths concluson was suggested by the results of Experment 1, and the concluson concernng the optmal level of face L* was confrmed drectly n Experment 2. The mages used n our experments contaned faces wth a wde varety of skn tones. Analyss of Caucasan and non-caucasan subgroups suggest that the conclusons concernng optmal face L* level may generalze to a wde array of face mages. We do note, however, that our mage sample was relatvely small and that follow-up work mght proftably probe the generalty of our results. For example, we do not know how senstve the data are to the nose propertes of the camera sensors. The analyss of the Experment 1 data by ethncty also suggests that the optmal global L* standard devaton for the rendered mage may depend on ethncty, although agan the generalty of ths result s not clear. Table 1 The overall percentage of tmes the output of each tonemappng method was chosen as the preferred mage n Experment 1. Results for each algorthm were obtaned by takng all of the parwse comparsons nvolvng the output of each algorthm and computng the percentage of tmes the output of that algorthm was chosen as preferred. Data were aggregated across all mages and observers. Images Clppng (%) Hstogram (%) Larson (%) Holm (%) All 26.4 19.0 19.0 35.7 Face 30.0 15.7 16.2 38.2 Nonface 18.8 25.9 24.9 30.4 5.2 Other Image Statstcs In addton to the mage tone characterstcs on whch we prevously reported n detal, we also examned other possble predctors of mage qualty. These ncluded chromatc varables and a hstogram dfference measure. The hstogram dfference measure ncreased wth the dfference between the lumnance hstogram of the nput and output of the tone-mappng algorthms. The chromatc varables dd not provde predctve power. Ths s perhaps not surprsng gven that the mages were all color balanced to a common llumnant and that the tone-mappng algorthms dd not affect pxel chromatctes. The hstogram dfference measure was correlated wth mage qualty for the face mages. Journal of Electronc Imagng 023003-10
Delahunt, Zhang, and Branard: Perceptual mage qualty... Fg. 9 Optmal value u face 0 for Experment 2. The plot shows the percent varance explaned by ũ face jk as a functon of the optmal value u face 0. Thn lne: full data set. Thck lne: data set when three mages wth artfacts were excluded. A stepwse regresson analyss, however, showed that addng the hstogram dfference measure to the face L* and mage L* standard devaton dd not explan substantal addtonal varance. Holm 16,38 has suggested that classfyng mages based on hstogram propertes and then applyng dfferent tone mappng dependng on the classfcaton can be effectve. To explore ths, we computed Holm s key value statstc from our nput mage hstograms and dvded the scenes nto two sets, low key and hgh key, based on ths statstc. Low-key scenes have lumnance hstograms that are skewed toward dark values, whereas hgh-key scenes have lumnance hstograms that are skewed toward lght values. In Experment 1, we found that the relaton between global L* value and mage qualty was strong for the low-key scenes and not sgnfcant for the hgh-key scenes, whereas the relaton between global L* standard devaton and mage qualty was sgnfcant for both low- and hgh-key scenes. There was a dfference n optmal global L* standard devaton between the two sets, but ths dfference was not stable wth respect to small perturbatons of the crteron key value used to dvde the data set. In Experment 2, the dependence on face L* values was sgnfcant for both low- and hgh-key scenes wth the optmal value varyng between 52 low-key and 47 hgh-key. Further experments focused on the stablty of scene key as a modulator of optmal tone characterstcs, as well as on other potental hgher-order hstogram statstcs e.g., degree of bmodalty, would be of nterest. 5.3 Relaton to Other Work The work here emphaszes comparsons are among mages dsplayed on a common output devce, so that the dynamc range of the comparson set s constant. Ths s a reasonable choce for the goal of mprovng the appearance of mages acqured wth current dgtal cameras, whose mage capture range s approxmately matched to current dsplay technology. In contrast, a number of papers have examned tonemappng across large changes n dynamc range between nput and output. 8,15,17,18,20,39 The expermental methods and analyss presented here are general and could be used to evaluate the effcacy of these methods for hgh-dynamc range magery. A second feature of our work s our focus on the tone characterstcs of the dsplayed mages, rather than on the functonal form of the tone-mappng curve. The results presented here suggest that there s consderable utlty n examnng tone characterstcs. Other recent expermental work 19,20 has focused on the effcacy of tone-mappng operators per se. These two approaches may be vewed as complementary. Also of note s the dverse set of psychophyscal technques that have been employed across studes. 19,20,39 Here we have focused on mage preference, whch s conceptually qute dfferent from perceptual fdelty. 5.4 Usng the Results Although our postve results only apply to mages that contan faces, such mages probably form a large proporton of those acqured by the average camera user many consumers take pctures of ther frends and famles. Thus our results have the potental for leadng to useful practcal algorthms. Snce our work shows how preference for mages contanng faces depends on tone varables, tone-mappng methods mght proftably nclude algorthms to dentfy mages that contan faces and to apply approprate mappng parameters to these mages. Face recognton software has advanced greatly n recent years. See recent revew by Pentland and Choudhury 40. Indeed, the present work led drectly to the development of a novel propretary tonemappng algorthm at Aglent Laboratores. 42 The dea that emprcal mage preference studes can enable development of effectve mage processng algorthms was also supported by our earler study. 4 We beleve further studes hold the promse of provdng addtonal algorthmc nsghts. Acknowledgments The authors wsh to thank Jerry Tetz for help wth mage acquston and Jack Holm and Jeff DCarlo for help wth mage processng and mplementaton of tone-mappng methods. Jack Holm also helped wth the experment room setup. Fnally they would lke to thank Russell lmura, Amnon Slversten, Joyce Farrell, and Yngme Lavn for helpful suggestons. References 1. C. J. Bartleson, Memory colors of famlar objects, J. Opt. Soc. Am. 50, 73 77 1960. 2. S. M. Newhall, R. W. Burnham, and J. R. Clark, Comparson of successve wth smultaneous color matchng, J. Opt. Soc. Am. 47 1, 43 56 1957. 3. R. M. Boynton, L. Fargo, C. X. Olson, and H. S. Smallman, Category effects n color memory, Color Res. Appl. 14, 229 234 1989. 4. P. Longere, X. Zhang, P. B. Delahunt, and D. H. Branard, Perceptual assessment of demosacng algorthm performance, Proc. IEEE 90, 123 132 2002. 5. H. de Rdder, Naturalness and mage qualty: saturaton and lghtness varaton n color mages of natural scenes, J. Imagng Technol. 40 6, 487 493 1996. Journal of Electronc Imagng 023003-11
Delahunt, Zhang, and Branard: Perceptual mage qualty... 6. E. A. Fedorovskaya, H. D. Rdder, and F. J. J. Blommaert, Chroma varatons and perceved qualty of color mages of natural scenes, Color Res. Appl. 22 2, 96 110 1997. 7. T. Tanaka, R. S. Berns, and M. D. Farchld, Predctng the mage qualty of color overhead transparences usng a color-appearance model, J. Electron. Imagng 6 2, 154 165 1997. 8. G. W. Larson, H. Rushmeer, and C. Patko, A vsblty matchng tone reproducton operator for hgh dynamc range scenes, IEEE Transactons on Vsualzaton and Computer Graphcs, avalable at http://radste.lbl.gov/radance/papers 1997. 9. C. N. Nelson, Tone and color reproducton Part 1: Tone Reproducton, The Theory of the Photographc Process, Macmllan, New York 1977. 10. C. J. Bartleson and E. J. Breneman, Brghtness reproducton n the photographc process, Photograph. Sc. Eng. 11 4, 254 262 1967. 11. C. J. Bartleson and E. J. Breneman, Brghtness percepton n complex felds, J. Opt. Soc. Am. 57, 953 957 1967. 12. L. D. Clark, Mathematcal predcton of photographc pcture qualty from tone-reproducton data, Photograph. Sc. Eng. 11 5, 306 315 1967. 13. R. G. W. Hunt, The effect of vewng condtons on requred tone characterstcs n colour photography, Brt. Knematogr. Sound Telev. 51, 268 275 1969. 14. R. W. G. Hunt, I. T. Ptt, and P. C. Ward, The tone reproducton of colour photographc materals, J. Photogr. Sc. 17, 198 204 1969. 15. E. Renhard, M. Stark, P. Shrley, and J. Ferwerda, Photographc tone reproducton for dgtal mages, ACM Trans. Graphcs 21, 267 276 2002. 16. J. Holm, Photographc tone and colour reproducton goals, CIE Expert Symp. 96 on Colour Standards for Image Technology, 51 56 1996. 17. J. Tumbln, J. K. Hodgns, and B. K. Guenter, Two methods for dsplay of hgh contrast mages, ACM Trans. Graphcs 18, 56 94 1999. 18. J. M. DCarlo and B. A. Wandell, Renderng hgh dynamc range scenes, Proc. SPIE 3965, 392 401 2000. 19. F. Drago, W. L. Martens, K. Myszkowsk, and H.-P. Sedel, Perceptual evaluaton of tone mappng operators wth regard to smlarty and preference, Report No. MPI-I-2002-4-002, Max-Planck-Insttut 2002. 20. G. J. Braun and M. D. Farchld, Image lghtness rescalng usng sgmodal contrast enhancement functons, J. Electron. Imagng 8, 380 393 1999. 21. J. L. Smonds, A quanttatve study of the nfluence of tonereproducton factors on pcture qualty, Photograph. Sc. Eng. 5 5, 270 277 1961. 22. S. B. Novck, Tone reproducton from colour telecne systems, Brt. Knematogr. Sound Telev. 51 10, 342 347 1969. 23. D. A. Slversten and J. E. Farrell, Effcent method for pared comparson, J. Electron. Imagng 10 2, 394 398 2001. 24. Kodak, Programmer s reference manual models: DCS-200c, DCS- 200m, DCS-200c, DCS-200m, Report 2, Eastman Kodak Company 1992. 25. P. L. Vora, J. E. Farrell, J. D. Tetz, and D. H. Branard, Image capture: smulaton of sensor responses from hyperspectral mages, IEEE Trans. Image Process. 10, 307 316 2001. 26. D. H. Branard, An deal observer for appearance: reconstructon from samples, Report No. 95-1, UCSB Vson Labs Techncal Report, Santa Barbara, CA 1995, http://color.psych.upenn.edu/branard/ papers/bayessamplng.pdf 27. D. H. Branard, Bayesan method for reconstructng color mages from trchromatc samples, IS&T 47th Annual Meetng, 375 379 1994. 28. D. H. Branard and D. Sherman, Reconstructng mages from trchromatc samples: from basc research to practcal applcatons, IS&T/SID Color Imagng Conf.: Color Scence, Systems, and Applcatons, 4 10 1995. 29. M. J. Vrhel, R. Gershon, and L. S. Iwan, Measurement and analyss of object reflectance spectra, Color Res. Appl. 19 1, 4 9 1994. 30. D. H. Branard and W. T. Freeman, Bayesan color constancy, J. Opt. Soc. Am. A 14 7, 1393 1411 1997. 31. W. Fre and C. C. Chen, Fast boundary detecton: a generalzaton and a new algorthm, IEEE Trans. Comput. 26, 988 998 1977. 32. D. H. Branard, Calbraton of a computer controlled color montor, Color Res. Appl. 14 1, 23 34 1989. 33. ISO, Vewng condtons for graphc technology and photography, ISO 3664, 1998 E 1998. 34. D. H. Branard, The psychophyscs toolbox, Spatal Vs. 10 4, 433 436 1997. 35. D. G. Pell, The Vdeo Toolbox software for vsual psychophyscs: transformng numbers nto moves, Spatal Vs. 10 4, 437 442 1997. 36. S. Ishhara, Tests for Colour-Blndness, Kanehara Shuppen Company, Ltd., Tokyo, Japan 1977. 37. CIE, Recommendatons on unform color spaces, color-dfference equatons, psychometrc color terms, Report No. Supplement No. 2 to CIE Publcaton No. 15, Bureau Central de la CIE 1978. 38. J. Holm, A strategy for pctoral dgtal mage processng, Proc. of the IS &T/SID 4th Color Imagng Conf., 194 201 1996. 39. A. MacNamara, Vsual percepton n realstc mage synthess, Comput. Graphcs Forum 20, 211 224 2001. 40. A. Pentland and T. Choudhury, Face recognton for smart envronments, Computer 33 2, 50 55 2000. 41. B. Efron and R. LePage, Introducton to bootstrap, Explorng the Lmts of Bootstrap, L. Bllard, Ed., Wley & Sons, New York 1992. 42. X. Zhang, R. W. Jones, I. Baharav, and D. M. Red, System and method for dgtal mage tone mappng usng an adaptve sgmodal functon based on perceptual preference gudelnes. U.S. Patent No. EP02000691 Jan. 2005. Peter Delahunt receved hs BS degree n psychology from Lancaster Unversty, England n 1996. He receved hs MA and PhD degrees n psychology from the Unversty of Calforna, Santa Barbara, n 1998 and 2001, respectvely. He s currently workng as a human factors scentst for Exponent Inc. Xueme Zhang receved her bachelor s degree n psychology from Bejng Unversty, master s degree n statstcs, and PhD n psychology from Stanford Unversty. She s currently a research scentst workng n Aglent Technologes Laboratores. Davd Branard receved hs AB n Physcs from Harvard Unversty n 1982. He attended Stanford Unversty for graduate school and receved hs MS degree n electrcal engneerng and PhD n psychology, both n 1989. He s currently professor of psychology at the Unversty of Pennsylvana. Hs research nterests nclude human vson, mage processng, and vsual neuroscence. Journal of Electronc Imagng 023003-12