Natural Image Denoising: Optimality and Inherent Bounds

Transcription

1 atural Image Denosng: Optmalty and Inherent Bounds Anat Levn and Boaz adler Department of Computer Scence and Appled Math The Wezmann Insttute of Scence Abstract The goal of natural mage denosng s to estmate a clean verson of a gven nosy mage, utlzng pror knowledge on the statstcs of natural mages. The problem has been studed ntensvely wth consderable progress made n recent years. However, t seems that mage denosng algorthms are startng to converge and recent algorthms mprove over prevous ones by only fractonal db values. It s thus mportant to understand how much more can we stll mprove natural mage denosng algorthms and what are the nherent lmts mposed by the actual statstcs of the data. The challenge n evaluatng such lmts s that constructng proper models of natural mage statstcs s a long standng and yet unsolved problem. To overcome the absence of accurate mage prors, ths paper takes a non parametrc approach and represents the dstrbuton of natural mages usng a huge set of 0 0 patches. We then derve a smple statstcal measure whch provdes a lower bound on the optmal Bayesan mnmum mean square error (MMSE). Ths mposes a lmt on the best possble results of denosng algorthms whch utlze a fxed support around a denosed pxel and a generc natural mage pror. Our fndngs suggest that for small wndows, state of the art denosng algorthms are approachng optmalty and cannot be further mproved beyond 0.dB values.. Introducton atural mage denosng s defned as the problem of estmatng a clean verson of a nose corrupted mage, gven a- pror knowledge that the orgnal unknown sgnal s a natural mage. The man dea n ths settng s that mage denosng can be performed usng not only the nosy mage tself but rather usng a sutable pror on natural mage statstcs. Image denosng algorthms have drastcally advanced over the last few decades [7, 7, 20, 2, 3, 0, 8]. Varous mage prors have been learned [20, 2, 26], and partcularly mpressng results have been obtaned wth non parametrc technques such as non local means [3] or BM3D [8], and sparse representaton methods such as KSVD [0]. However, t seems that the performance of denosng algorthms s startng to converge. Recent technques typcally mprove over prevous ones by only fractonal db values. In some cases the dfference between the results of competng algorthms s so small and nconclusve, that one actually has to successvely toggle between mages on a montor to vsually compare ther denosng qualty. Ths rases the queston of whether the error rates of current denosng algorthms can be reduced much further, or whether there are nherent lmtatons mposed by the statstcal structure of natural mages? The goal of ths paper s to derve a lower bound on the best possble denosng error under a well defned statstcal framework. Such a bound can help us understand f there s hope to sgnfcantly mprove the current stateof-the-art mage denosng wth even better algorthms, or whether we have nearly approached the fundamental lmts. Understandng the lmts of natural mage denosng s also mportant as an nstance of a more fundamental computer and human vson challenge: modelng the statstcs of natural mages and understandng the nherent lmts of ther statstcal power. Several works attempted to estmate the entropy of natural mages [5, 4]. However, there s no drect relaton between entropy and our ablty to solve low level vson tasks. Furthermore, numerous research attempts have been devoted to the learnng of natural mage prors [27, 2, 2, 26, 8, 2]. However, t s not clear f these models actually capture the full statstcal structure of natural mages. In prncple, a perfectly accurate natural mage pror would allow us to compute optmal Bayesan estmators for low level vson tasks such as denosng, superresoluton, deconvoluton and others. Therefore, understandng how far from optmalty do current denosng algorthms stand, wll gve us an ndcaton of how far are we from understandng and modelng the statstcal structure of natural mages. Several works studed the lmts of mage denosng. Some methods focused mostly on SR arguments [3, 24, 23] wthout takng nto account the strength of natural mage prors. Smlarly, lmts on the related problem of superresoluton have been derved n the past [, 6], but they agan account for the numercal stablty of the lnear system beng nverted, and do not model the addtonal gan provded by natural mage prors. In fact, n the same paper, 2833

2 Baker and Kanade [] show that ther super-resoluton lmts can be broken usng a face class pror. In the statstcal lterature, sharp bounds on mage denosng and edge detecton have been obtaned. However, these bounds assume over-smplfed mage models, such as pecewse constant regons separated by sharp edges wth smooth boundares [9, 4]. The non local means denosng algorthm [3] s proven to be asymptotcally optmal gven an nfntely large mage. However, ts expected error and devaton from optmalty on realstc fnte-sze mages s unclear. Recently, Chatteree and Mlanfar [5, 6] derved denosng bounds that do account for natural mage statstcs. Ther model assumes that the mage patches can be factorzed nto a small number of clusters, and each such cluster can be descrbed usng second order statstcs. Ths, however, s a strong assumpton whch can largely affect the conclusons. The lack of a more detaled analyss whch accounts for the statstcs of natural mages s due to the fact that modelng the statstcs of natural mages s an extremely hard problem, wth no agreeably good model suggested up to ths day. Thus, any evaluaton whch would rely on one of the exstng mage prors wll be strongly based by the naccuracy of the pror. In order to overcome ths challenge, our am n ths paper s to estmate a lower bound on mage denosng wthout usng any partcular parametrc model. Usng the sum of square dfferences error metrc, we derve a lower bound on the smallest possble reconstructon error of denosng algorthms whch consder a k k pxels support around a nosy pxel, and use a generc natural mage pror on k k patches. Thus, the possble gan by mage specfc prors [3, 0, 8], rather than generc ones, s beyond the scope of ths analyss. Our approach bulds on the success of recent large mage databases approaches n hgh level vson and graphcs applcatons [, 22], and represents the dstrbuton of natural mages n a non-parametrc way usng a huge set of 0 0 patches. We use ths patch set to approxmate the optmal Bayesan mnmum mean squared error (MMSE) estmator and ts expected error. Whle every fnte set of samples provdes only an approxmaton for the actual MMSE, we derve statstcal formulas whch, under certan condtons, allow us to compute both a lower bound and an upper bound on the MMSE. We show that for small support sze k or for large nose varance, the number of patches (0 0 ) s large enough. Hence, the lower and upper bounds concde and we can estmate the actual MMSE. At the more challengng cases of very large patch szes or very small nose levels, we only get a lower bound on the best possble denosng error. Our calculatons suggest that for the tested support szes, the state of the art denosng results of BM3D [8] are already close to optmalty, and cannot be further mproved beyond 0. db values. On the other hand, ncreasng the support sze does carry some potental for mproved denosng performance. However, ths ncrease should probably requre swtchng to parametrc approaches, snce non-parametrc approaches are nherently lmted by the densty of nearest neghbors, whch drastcally decreases wth wndow sze. Developng parametrc models whch can acheve state of the art denosng results s also useful snce they can be appled to other low-level vson tasks, and more mportantly, because ther generalzaton power can mprove our understandng of natural mages. 2. Boundng denosng performance 2.. Problem formulaton In mage denosng, one s gven a nosy verson y of a clean mage x y = x + n () and the am s to estmate a cleaner verson ŷ from y. In ths paper, we consder random nose vectors n whose entres are dstrbuted accordng to a zero mean..d. Gaussan wth varance σ 2. We restrct the dscusson to denosng algorthms whch denose each pxel separately usng a k k pxels support around t. Equvalently, we assume that x, y are k k patches and focus on the estmaton of the central pxel n each patch x, whch we denote by x c. The success of mage denosng algorthms s usually evaluated by PSR values, whch essentally measure the mean squared error (MSE) n reconstructon, averaged over a set of M test patches {(x, y )} M = «PSR = 0 log 0, MSE MSE M X (x,c ŷ,c) 2. (2) As dscussed n [25], the MSE may not provde an ultmate vsual qualty predcton. onetheless, we adopt the MSE for two reasons: a) t s the classcal measure beng optmzed by most exstng denosng algorthms, and b) t s correlated, though not perfectly, wth vsual qualty. Therefore, a lmt on MSE denosng s a reasonable proxy on how much we can mprove vsual qualty. The study of lower lmts correspondng to other qualty measures s an nterestng future research problem. Snce each nosy patch y could actually be generated by multple latent patches x, the problem s fundamentally ambguous and one cannot hope for a zero reconstructon error. Our goal s to understand what s the lowest possble MSE achevable by any denosng algorthm based on k k patches, gven a-pror knowledge that the nput mage has been randomly sampled from the set of natural mages. To ths end, we denote by p(x) the densty of k k patches of natural mages, and by p(y) the resultng densty of k k nosy patches. When context requres, we use the notaton p σ (y) to emphasze the explct dependence of the densty of nosy y patches on the nose level σ. The Gaussan..d. nose assumpton s not a hard one and the technque can be appled wth many other nose models. The non parametrc approach mostly requres the ablty to compute p(y x). 2834

3 2.2. Bayesan Mnmum Mean Square Error There are two equvalent nterpretatons for the MSE the error nterpretaton and the varance nterpretaton. In the error vew, we randomly sample clean patches x from p(x), add nose to generate y, denose y and measure the reconstructon error (x,c ŷ,c ) 2. The average of ths reconstructon error s Z Z MSE = p(x) p(y x)(x c ŷ c) 2 dydx (3) An equvalent nterpretaton of the MSE, whch wll serve us below, s the varance nterpretaton start from a nosy patch y, and measure the varance of p(x y) around t. That s, compute the sum of weghted dstances between ŷ and all possble x explanatons: Z Z MSE = p(y) p(x y)(x c ŷ c) 2 dxdy (4) Eq. (4) s obtaned from Eq. (3) by swtchng the order of ntegraton and applyng Bayes rule. The advantage of the varance vew s that from t, one can easly derve the optmal estmator, namely the Bayesan mnmum mean squared error (MMSE) estmator [3], whch s smply gven by the condtonal mean µ(y) = E[x c y] = p(x y)x c dx (5) = p(y x) p(y) p(x)x cdx The MMSE at any fxed y s then the condtonal varance, Z V(y) = E[(x c µ(y)) 2 y] = p(x y)(x c µ(y)) 2 dx (6) whereas the overall MMSE s MMSE = E[V(y)] = p(y)v(y)dy. (7) In the framework of natural mage denosng, the MMSE n (7) s the lowest achevable denosng error by any denosng algorthm. To compute ths MMSE and the correspondng optmal estmator µ(y), we need access to the true natural mage densty p(x). However, as dscussed n the ntroducton, an accurate model for ths densty s a long standng research problem [20, 2, 26, 9]. One opton would be to ft some parametrc model q(x), for example, a feld of experts model [2] or a Gaussan mxture model. We can then compute an approxmated mean ˆµ(y) = /q(y) p(y x)q(x)x c dx. However, snce q(x) s only an approxmaton to the true natural mage densty p(x), the error of the estmator ˆµ(y), lke the error of any denosng algorthm, s only an upper bound on the MMSE. The dfference n MSE depends on the qualty of the approxmaton, whch s not easy to evaluate, n partcular snce p(x) s unknown. Our goal, n contrast, s to estmate the MMSE wthout commttng to any specfc approxmate parametrc model, or essentally, wthout even explctly knowng p(x). The key dea s that even though p(x) s unknown, we are stll able to sample from t. Therefore we consder a non-parametrc representaton of the dstrbuton, usng a large set of about = 0 0 natural mage patches. Ths allows us to approxmate the ntegral of Eq. (5) by averagng over samples: ˆµ(y) = where for Gaussan nose p(y x) = P p(y x)x,c P p(y x), (8) x y 2 (2πσ 2 e 2σ ) d/2 2 (9) and d = k 2. As, ˆµ(y) ndeed converges to the true MMSE estmator. However, for any fnte set, Eq. (8) s only an approxmaton, and thus, lke any other denosng algorthm, ts average error provdes an upper bound on the MMSE. However, our goal s to bound the MMSE from below, stll, wthout knowng p(x) explctly, but havng at our dsposal samples from t. The man dea n dervng both an upper and a lower bound on the MMSE, s to use the two MSE formulatons n Eqs. (3) and (4). Gven a set of M clean and nosy pars {( x, y )} M = and another ndependent set of clean patches {x } =, both randomly sampled from natural mages, we compute MMSE U = (ˆµ(y ) x,c ) 2 (0) M MMSE L = ˆV(y ) () M where ˆV(y ) s the approxmated varance: ˆV(y ) = P p(y x)(ˆµ(y) x,c)2 P p(y x) (2) MMSE U and MMSE L are both random varables whch depend on the partcular set of x, y samples. When the sample sze approaches nfnty, they converge to the exact MMSE. However, we show that for a fnte sample, n expectaton, MMSE U and MMSE L provde upper and lower bounds on the best possble MMSE. ote the key dfference between these two quanttes: MMSE U uses explct knowledge of the orgnal nose-free patch x, whle MMSE L does not nvolve t. Snce MMSE U bascally measures the error of the estmator ˆµ(y ), t provdes, lke every denosng algorthm, an upper bound on the MMSE. The term MMSE L s analyzed n Sec When s suffcently large, the two computed values MMSE U and MMSE L are smlar, and we get an accurate estmate for the actual optmal MMSE. As we show n secton 3 ths happens when the patch sze k s small, or when 2835

4 the nose varance σ 2 s hgh, snce n such cases there s a large number of vald nearest neghbors around each patch. In the harder cases, we cannot compute the MMSE exactly, but MMSE L stll provdes a lower bound on the best denosng results we can expect The sample varance as a lower bound on the MMSE Our goal s to show that the approxmated varance estmated from a fnte set of samples (Eq. (2)), provdes a lower bound on the correct varance (Eq. (6)). To see ths at an ntutve level, consder the numerator of Eq. (2). Snce ˆµ(y) s the weghted mean of {x } =, t mnmzes the mean squared dstance from these patches, and n partcular, t acheves a smaller squared dstance compared to the exact unknown mean µ(y): p(y x )(x,c ˆµ(y )) 2 p(y x )(x,c µ(y )) 2 Thus, n expectaton " # X E p(y x )(x,c ˆµ(y )) 2 " # X E p(y x )(x,c µ(y )) 2 = p(y )V(y ), (3) where E [ ] denotes expectaton wth respect to all possble sets of..d. patches from p(x). The denomnator of Eq. (2) s also a random varable whose expectaton s p(y ), thus, roughly, E [ˆV(y)] V(y). However, snce both numerator and denomnator are random varables, a more careful analyss of ˆV(y) s needed. The followng clam derves a second order approxmaton to E [ˆV(y)] for a fxed patch y. Clam Asymptotcally, as the tranng set sze, the expected value of the approxmated varance defned n Eq. (2) s «E [ˆV(y)] = V(y) + C(y)B(y) + o, (4) wth B(y) = E σ [x 2 c y] 3E σ [x c y] 2 2E σ [x 2 c y] +4E σ [x c y]e σ [x c y] (5) C(y) = p σ (y) (4πσ 2 ) d/2 p σ (y) 2 (6) where p σ, E σ [ ], p σ, E σ [ ] denote probablty and expectaton of random varables wth nose standard dervaton σ and σ respectvely. σ s the actual standard devaton and σ = σ/ 2. The proof of Clam s provded n the appendx. It uses an asymptotc expanson of ˆV(y) n, and neglects hgh V(y) 4 x Correct Varance Approx. Varance, =5 Approx. Varance, =20 Approx. Varance, =00 σ y 0 σ y y B(y ) x (a) (b) Fgure. Evaluatng the bas of ˆV(y). (a) For a D Gaussan densty p(x), we compare the expected sample varance E [ˆV(y)] to ts exact value V(y), whch s constant n y for a Gaussan dstrbuton. As predcted theoretcally, E [ˆV(y)] V(y), wth the bas decreasng wth ncreasng number of samples or at hgh densty regons. (b) Plug-n bas estmates ˆB(y) of real natural mage patches, sorted n ascendng order. ote that up to small numercal errors, the bas estmates of all patches are negatve. order terms. It s hence vald for suffcently large and suffcently common y. ext, we consder a local Gaussan approxmaton for p(x) around x = y, e.g., a Laplace approxmaton consstng of a second order Taylor expanson of lnp(x). For a Gaussan dstrbuton we can analytcally compute the leadng order bas term C(y)B(y) from Eqs. (5) and (6). The followng clam, proven n the appendx, shows that B(y) s negatve for all y values. Snce C(y) > 0, t follows that E [ˆV(y)] V(y). Thus, n expectaton, MMSE L provdes a lower bound on the MMSE, as we am to show. Clam 2 For a Gaussan dstrbuton, B(y) 0 for all y. Fgure (a) consders a smple case of a D Gaussan dstrbuton and compares the exact varance V(y) wth E [ˆV(y)], averaged over 0, 000 realzatons of sets of = 5, 0 or 5 samples. The bas s always negatve and E [ˆV(y)] V(y). As evdent from Eq. (6), C(y) s large when p(y) s small, and so for a Gaussan dstrbuton, the bas ncreases exponentally wth y. The densty of natural mage patches s not Gaussan, so a Laplace approxmaton mght be naccurate and Clam 2 may not drectly apply. To evaluate ths on real data, we used a plug-n estmator ˆB(y) for the term B(y). That s, we consdered M = 2, 000 nosy 3 3 patches {y } wth nose varance σ = 8. For each y the expectatons n Eq. (5) were estmated by averagng over an ndependent large set of clean patches {x }. Fgure (b) shows the resultng ˆB(y ) values, sorted n ascendng order for vsualzaton purposes. As seen n the plot, the estmated values ˆB(y ) are all negatve up to small numercal errors. 3. Experments To evaluate the MMSE we use a set of 20, 000 mages from the LabelMe dataset [22]. We thus mplctly consder ths dataset an unbased representatve of nose-free 2836

5 PSR MMSE U MMSE L LMMSE BM3D KSVD GSM Support sze k PSR MMSE U MMSE L LMMSE BM3D KSVD GSM Support sze k PSR MMSE U MMSE L LMMSE BM3D KSVD GSM Support sze k (a) σ = 8 (b) σ = 55 (c) σ = 70 Fgure 2. PSR values of several recent denosng algorthms along wth our MMSE lower and upper bounds. As predcted by the theory, the performance of all algorthms are bounded by our MMSE L estmate, although BM3D approaches the bound by fractonal db values. (ote that snce PSR = 0log 0(MSE), the MMSE lower bound turns nto an upper bound on the best achevable PSR). natural mage statstcs. 2 To avod quantzaton and JPEG artfacts we frst low-passed all mages and down-sampled them by a factor of two. These mages provde a set of about = 0 0 natural mage patches {x }. We use another M = 2, 000 clean and nosy pars of patches {( x, y )}, denose them by computng the weghted mean of Eq. (8), and measure the mean squared error MMSE U (Eq. (0)), and the lower bound MMSE L (Eq. ()). Ths s an ntensve computaton, whch took about a week of computaton on a 00 CPUs cluster. In Fgure 2 we compare the PSR values of varous denosng algorthms to MMSE U and MMSE L wth three nose levels σ = 8, 55, 70, for mage ntenstes n the [0, 255] range. We plot our bounds as a functon of the wndow sze k. We tred to match the support sze of the tested algorthms as well. We used the BM3D [8] algorthm wth patch szes of 4, 8 and 2 pxels (for 8, 2 pxels we used the authors parameters, and for 4 pxels, our adaptaton). We assocate the KSVD [0] and Portlla et al. GSM[20] algorthms wth the largest support sze k consdered, snce these are global denosng algorsms that do not consder fnte support patches. As a baselne comparson we also present a lnear mnmum mean square error (LMMSE) estmator [3]. Ths estmator, also known as the Wener flter, uses only the second order statstcs of the data, by fttng a sngle k 2 dmensonal Gaussan to the set of mage patches. When computng our MMSE U and MMSE L scores we used the bootstrappng method to evaluate the varance, by drawng multple patch subsets. The standard devaton of the estmaton s rather small, rangng from 0.05dB for small k values to 0.2dB at the large ones. The results n Fgure 2 suggest several observatons whch we dscuss below. 2 Even f these nput mages do contan a small amount of nose, t s neglgble w.r.t. the nose level added n our experments. If nonetheless, these patches do contan a small amount of nose, then the goal s formulated as the reconstructon of sgnals wth ths slghtly perturbed dstrbuton, whch s stll a statstcally well defned problem. Tghtness of bounds: For small wndow szes or hgh nose, the upper and lower bounds concde and hence we have obtaned an accurate estmate of the exact optmal MMSE value. For harder cases, there s a gap between the upper and lower bounds. Whle we are unable to estmate the exact MMSE, we stll get a vald lower bound on t. Ths happens when there are too few samples at a normalzed dstance of σ 2 from the nosy patch. To demonstrate that, we measured the number of nearest neghbors wthn normalzed dstance of one standard devaton from each nosy patch. Under the assumpton of Gaussan nose the dfference y x 2 follows a χ 2 dstrbuton wth k degrees of freedom. Therefore, for each nosy patch y we consder all patches at a squared dstance of σ 2 ( + 2/k 2 ): n = # x k 2 x y 2 < σ 2 + «ff 2 k (7) In Fgure 3 we plot the cumulatve dstrbuton of n values, for two wndow szes k = 3 and k = 9, both wth σ = 8. We see that for small k, most patches have a large number of nearest neghbors, whereas for k = 9, 3% of the patches have zero neghbors. Accordngly, at k = 9, σ = 8, there s a gap between MMSE U and MMSE L n Fg 2(a). ear optmalty of state of the art denosng results: As predcted by the theory, the emprcal errors of all denosng algorthms are larger than our lower bound. onetheless, for most cases the PSR values of BM3D are wthn 0.dB of the optmal ones, suggestng that the BM3D results are qute close to optmalty, for small patch szes. Thus, future denosng algorthms that use small patches have very lttle room for mprovement over BM3D 3. However, whle 0.dB seems lke a very small mprovement, t may stll be vsually notceable. 3 Ths does not drectly mply that BM3D cannot be further mproved, snce whle BM3D s a patch based algorthm, t utlzes a wder support by lookng for neghbors n the entre mage. 2837

6 00 00 Examples percent, h(n) Examples percent, h(n) umber of neghbors, n umber of neghbors, n (a) k = 3 (b) k = 9 Fgure 3. Cumulatve dstrbuton functon of the number of neghbors wthn dstance σ 2 ( + 2/k), computed at σ = 8. For 3 3 patches, 99% of the examples had more than 2, 000 neghbors. In contrast, for a 9 9 patch sze, 3% of the examples had no neghbors wthn ths dstance. (a) Orgnal mage (b) osy nput Support sze: It seems that ncreasng the support sze carres some potental for ncreasng the PSR. Due to the curse of dmensonalty, non-parametrc technques suffer from an unavodable tradeoff between patch sze and the number of vald nearest neghbors, and hence are unlkely to be applcable for large wndow szes. Developng algorthms whch utlze a larger support sze thus probably requres swtchng from non-parametrc approaches to parametrc ones. Unfortunately, at the moment parametrc denosng algorthms are far behnd the non parametrc ones. (c) Opt. MMSE, PSR=23.93dB (d) BM3D, PSR=23.86dB Denosng at extreme nose levels: In Fg 2(c), the nose level s σ = 70. At ths hgh nose level, the mage pror results are very smlar to the results of a smple lnear mnmum mean square error (LMMSE) estmator whch utlzes only the second order statstcs of the data. Ths happens snce at hgh nose levels the sgnal content s lost and all the pror can do s to estmate a flat mage. The square error of such an estmator s proportonal to the varance of the data, therefore natural mage prors or Gaussan prors utlzng second order statstcs gve smlar results. The other extreme that we dd not evaluate here s very low nose. For low nose the effect of the pror s small and the error s proportonal to the nose varance. Therefore, natural mage prors have an nterestng effect only at medum nose levels. Vsual results: In Fgure 4 we used the Peppers mage at half resoluton to vsualze the denosng results of several recent denosng algorthms compared wth our approach, whch s the best possble denosng wth a generc natural mages pror. We used σ = 75 and 2 2 wndows for our approach and for BM3D. umercally, our result outperforms BM3D by 0.07dB, and ths dfference leads to a slghtly better vsual qualty. The vsualzaton of optmal denosng n Fg 4(c) s only a proof-of-concept, but not a practcal denosng algorthm- denosng ths mage requred two weeks of computaton on 00 CPUs. However, one can thnk of several ways to accelerate the (d) KSVD, PSR=22.4dB (e) GSM, PSR=23.28dB Fgure 4. Vsual comparson of our optmal MMSE and other algorthms, for σ = 75. The optmal MMSE and BM3D use 2 2 wndows. Our approach acheves slghtly hgher PSR comparng to BM3D, and a somewhat better vsual qualty. search wth approxmated nearest neghbor technques and a smarter encodng of the large set of 0 0 mage patches. Image specfc bounds: Our analyss s based on the knowledge of a generc mage pror. An nterestng queston for future research, as recently consdered by [5, 6], s whether there s a sgnfcant advantage n adaptng the pror to the specfc statstcs of the observed mage, as done by some recent denosng algorthms [3, 0, 8]. In fact, such algorthms can be thought of as nstances of a generc pror whose support sze s the entre mage, snce the support sze essentally means that an algorthm can see and use n the estmaton all pxels wthn a k k wndow around a nosy pxel. Unfortunately, we cannot evaluate tght bounds at very large wndow szes. 2838

7 4. Dscusson Ths paper derved a statstcal measure for the best possble denosng results utlzng a generc natural mage pror. Our fndngs suggest that for small wndows, state of the art denosng algorthms cannot be further mproved beyond fractonal db values. Consderng a wder support carres some potental for mproved results. However, ncreasng the support sze of non-parametrc algorthms mght lead to a dead-end, and parametrc approaches are requred. Unfortunately, at the moment parametrc algorthms perform well behnd non-parametrc ones. Developng parametrc algorthms whch can acheve state of the art results s also mportant because they can generalze for other lowlevel vson tasks. The evaluaton methodology used here can be easly appled to other low-level vson problems such as superresoluton, deconvoluton, or npantng. However, some of these problems, such as the deconvoluton of a wde support kernel, rely on a large support of pxels. Hence we may not be able to fnd a suffcent number of nearest neghbors for a tght lower bound estmaton. Yet, understandng the lmts of mage denosng s mportant as a step toward other lowlevel vson problems, and moreover, as a step toward an understandng of the nherent lmts of natural mage statstcs. Acknowledgments: The authors thank the ISF, BSF and ERC for provdng fnancal support. 5. Appendx Proof of Clam : For future use we denote by A(σ, k) = (4πσ 2 ) k2 /2 and recall that σ = σ/ 2. The frst step n the proof s to nsert Eq. (8) nto Eq. (2), whch yelds the followng more convenent expresson for ˆV(y), p(y x )x ˆV(y) 2,c = ( p(y x )x,c ) 2 p(y x ) ( p(y x )) 2 = A A 2. (8) The man dea s to analyze the bas of each of these terms separately, whereby for each term we further consder the mean and fluctuatons of ther numerator and denomnator. For the term A we shall use the followng equaltes E[p(y x)] = R p(x)p(y x)dx = p σ(y) E[p(y x)x 2 c] = p σ(y)e σ[x 2 c y] (9) where p σ (y) s the densty of y-patches at nose level σ, and E σ [ y] denotes expectaton wth respect to p σ (y). The two expressons n Eq.(9) are nothng but the mean of the denomnator and numerator of A, respectvely. We thus rewrte the term A as ( p σ (y)e[x 2 c y] + A = ( p σ (y) + p(y x)x ) 2,c pσ(y)e[x2 c y] p σ(y)e[x 2 c y] p(y x) p σ(y) p σ(y) ) (20) ext, we assume and that the patch y s not too rare, such that p(y x ) p σ (y) p σ (y) Then, usng a Taylor expanson for small ǫ, + ǫ = ǫ + ǫ2 + O(ǫ 3 ) we obtan the followng asymptotc expanson for the frst term, (where for ease of notaton we wrte p(y) for p σ (y))! A E[x 2 c y] + X p(y x )x 2,c p(y)e[x 2 c y] p(y)e[x 2 c y]! X p(y x ) p(y) + 2! X p(y x ) p(y) p(y) p(y) (2) We now take the expectaton of Eq. (2) over x samples. We use the fact that E[p(y x) p(y)] = 0, E[p(y x)x 2 c p(y)e[x 2 c y]] = 0. We also neglect all O(/2 ) terms. E [A ] = E[x 2 c y] + E[p(y x) 2 ] p(y) 2 E[p(y x) 2 x 2 c] p(y) 2 E[x 2 c y] + o(/) Smlarly, for the second term A 2 we use the fact that «(22) E[p(y x) 2 ] = R p(x) e x y 2 /σ 2 dx = A(σ, k)p σ (y) (2πσ 2 ) k2 E[p(y x) 2 x c] = A(σ, k)p σ (y)e σ [x c y] E[p(y x) 2 x 2 c] = A(σ, k)p σ (y)e σ [x 2 c y] and hence rewrte the second term A 2 as + A 2 = ˆµ(y) 2 = E[x c y] 2 + = E[x c y] 2 P + 2 p(y x )x,c p(y)e[x c y] p(y)e[x c y] + E[x c y] P + 2 p(y x ) p σ(y) p σ(y) + P p(y x )x,c p(y)e[x c y] p(y)e[x c y] + P 2 p(y x ) p σ(y) p σ(y) + 3 P p(y x )x,c p(y)e[x 2 c y] p(y)e[x c y] P 2 p(y x ) p σ(y) p σ(y) (23) P «p(y x )x,c p(y)e[x 2 c y] p(y)e[x c y] «2 P p(y x ) p σ(y) p σ(y) P «p(y x )x,c p(y)e[x 2 c y] p(y)e[x c y] P «2 p(y x ) p σ(y) p σ(y) (24) Takng expectatons and omttng O(/ 2 ) terms, we get: E [A 2] = E[x c y] 2 4 E[p(y x) 2 x c] p(y) 2 E[x + E[p(y x) 2 x 2 c] c y] p(y) 2 E[x c y] «E[p(y x) 2 ] + o(/) p(y) 2 (25) 2839

8 Substtutng the terms from Eq. (23) n Eqs. (22) and (25) yelds the desred Eq. (4). Proof of Clam 2: We note that B(y) (Eq. (5)) can be wrtten as: B(y) = V σ[x c y] 2V σ [x c y] 2(E σ [x c y] E σ[x c y]) 2 (26) where V[x c y] = E[x 2 c y] E[x c y] 2 denotes varance. Thus, t s suffcent to show that V σ [x y] 2V σ [x y] 0. We denote by Φ the covarance matrx of p(x) and by Ψ σ, Ψ σ the covarance matrces of p σ (x y), p σ (x y): ««Ψ σ = σ 2I d + Φ Ψ σ = σ 2I d + Φ, (27) where I d s the d dmensonal dentty matrx. The varance at x c s the (c, c) entry of these matrces: V σ [x y] = Ψ σ (c, c), V σ [x y] = Ψ σ (c, c). We denote by {λ l }, {γ l }, {γl } the egenvalues of the matrces Φ, Ψ σ, Ψ σ respectvely. We note that all these matrces are dagonal n the same bass and we can wrte Ψ σ(c, c) = X l u 2 lγ l, Ψ σ (c, c) = X l u 2 lγ l, (28) (u l s the c entry of egenvector l). We can also relate the egenvalues of Ψ σ, Ψ σ to the egenvalues of the uncondtonal covarance Φ: γ l = (λ l + σ 2 ), γ l = (λ l + (σ ) 2 ). (29) A smple calculaton shows that for every l: «γ l 2γl = λ l σ 2 λ l + σ 2 0 (30) 2 2λ l + σ 2 Usng Eqs. (30) and (28) we conclude: References Ψ σ(c, c) 2Ψ σ(c, c) = X l u 2 l(γ l 2γ l ) 0. (3) [] S. Baker and T. Kanade. Lmts on super-resoluton and how to break them. PAMI, [2] A. Bell and T. Senowsk. The ndependent components of natural scenes are edge flters. Vson Res., 997. [3] A. Buades, B. Coll, and J. Morel. A revew of mage denosng methods, wth a new one. Multscale Model. Smul., [4] D. Chandler and D. Feld. Estmates of the nformaton content and dmensonalty of natural scenes from proxmty dstrbutons. J. Opt. Soc. Am., [5] P. Chatteree and P. Mlanfar. Is denosng dead? IEEE Trans Image Processng, 200. [6] P. Chatteree and P. Mlanfar. Practcal bounds on mage denosng: From estmaton to nformaton. IEEE Trans Image Processng, 20. [7] R. Cofman and D. Donoho. Translaton-nvarant denosng. In Wavelets and Statstcs, 995. [8] K. Dabov, A. Fo, V. Katkovnk, and K. Egazaran. Image denosng by sparse 3-d transform-doman collaboratve flterng. IEEE Trans Image Processng, [9] J. Echhorn, F. Snz, and M. Bethge. atural mage codng n v: How much use s orentaton selectvty? PLoS Comput Bol, [0] M. Elad and M. Aharon. Image denosng va sparse and redundant representatons over learned dctonares. IEEE Trans Image Processng, [] J. Hays and A. Efros. Scene completon usng mllons of photographs. SIGGRAPH, [2] A. Hyvarnen, J. Hurr, and P. Hoyer. atural Image Statstcs A probablstc approach to early computatonal vson. Sprnger-Verlag, [3] Steven M. Kay. Fundamentals of statstcal sgnal processng: estmaton theory. Prentce-Hall, Inc., 993. [4] A. Korostelev and A. Tsybakov. Mnmax Theory of Image Reconstructon. Sprnger-Verlag, ew York, 993. [5] A. Lee, K. Pedersen, and D. Mumford. The nonlnear statstcs of hgh-contrast patches n natural mages. IJCV, [6] Z. Ln and H. Shum. Fundamental lmts of reconstructonbased superresoluton algorthms under local translaton. PAMI, [7] S. Osher, A. Sole, and L. Vese. Image decomposton and restoraton usng total varaton mnmzaton and the h norm. Multscale Model. Smul., [8] S. Osndero and G. Hnton. Modelng mage patches wth a drected herarchy of markov random felds. IPS, [9] J. Polzehl and V. Spokony. Image denosng: Pontwse adaptve approach. Annals of Statstcs, 3:30 57, [20] J. Portlla, V. Strela, M. Wanwrght, and E. Smoncell. Image denosng usng scale mxtures of gaussans n the wavelet doman. IEEE Trans Image Processng, [2] S. Roth and M.J. Black. Felds of experts: A framework for learnng mage prors. In CVPR, [22] B. Russell, A. Torralba, K. Murphy, and W. Freeman. Labelme: a database and web-based tool for mage annotaton. IJCV, [23] T. Trebtz and Y. Schechner. Recovery lmts n pontwse degradaton. In ICCP, [24] M. Unser, B. Trus, and A. Steven. A new resoluton crteron based on spectral sgnal-to-nose ratos. Ultramcroscopy, 987. [25] Z. Wang, A. C. Bovk, H. R. Shekh, and E. P. Smoncell. Image qualty assessment: From error vsblty to structural smlarty. IEEE Trans. on Image Processng, [26] Y. Wess and W. T. Freeman. What makes a good model of natural mages? In CVPR, [27] S. Zhu and D. Mumford. Pror learnng and gbbs reactondffuson. PAMI,