10-1 MMSE Estimation S. Lall, Stanford

Transcription

1 0 - MMSE Estimation S. Lall, Stanford MMSE Estimation Estimation given a pdf Minimizing the mean square error The minimum mean square error (MMSE) estimator The MMSE and the mean-variance decomposition Example: uniform pdf on the triangle Example: uniform pdf on an L-shaped region Example: Gaussian Posterior covariance Bias Estimating a linear function of the unknown MMSE and MAP estimation

2 0-2 MMSE Estimation S. Lall, Stanford Estimation given a PDF Suppose x : Ω R n is a random variable with pdf p x. One can predict or estimate the outcome as follows Given cost function c : R n R n R pick estimate ˆx to minimize E c(x, ˆx) We will look at the cost function c(x, ˆx) = x ˆx 2 Then the mean square error (MSE) is E ( x ˆx 2) = x ˆx 2 p x (x) dx

3 0-3 MMSE Estimation S. Lall, Stanford Minimizing the MSE Let s find the minimum mean-square error (MMSE) estimate of x; we need to solve min ˆx E ( x ˆx 2) We have E ( x ˆx 2) = E ( (x ˆx) T (x ˆx) ) = E ( x T x 2ˆx T x + ˆx T ˆx ) = E x 2 2ˆx T E x + ˆx T ˆx Differentiating with respect to ˆx gives the optimal estimate ˆx mmse = E x

4 0-4 MMSE Estimation S. Lall, Stanford The MMSE estimate The minimum mean-square error estimate of x is ˆx mmse = E x Its mean square error is E ( x ˆx mmse 2) = tracecov(x) since E ( x ˆx mmse 2) = E ( x E x 2)

5 0-5 MMSE Estimation S. Lall, Stanford The mean-variance decomposition We can interpret this via the MVD. For any random variable z, we have E ( z 2) = E ( z E z 2) + E z 2 Applying this to z = x ˆx gives E ( x ˆx 2) = E ( x E x 2) + ˆx E x 2 The first term is the variance of x; it is the error we cannot remove The second term is the bias of ˆx; the best we can do is make this zero.

6 0-6 MMSE Estimation S. Lall, Stanford The estimation problem Suppose x, y are random variables, with joint pdf p(x, y). We measure y = y meas. 3 2 We would like to find the MMSE estimate of x given y = y meas. 0 2 The estimator is a function φ : R m R n We measure y = y meas, and estimate x by ˆx est = φ(y meas ) We would like to find the function φ which minimizes the cost function J = E ( φ(y) x 2)

7 0-7 MMSE Estimation S. Lall, Stanford Notation We ll use the following notation. p y is the marginal or induced pdf of y p y (y) = p(x, y) dx p y is the pdf conditioned on y p y (x, y) = p(x, y) p y (y)

8 0-8 MMSE Estimation S. Lall, Stanford The MMSE estimator The mean-square-error conditioned on y is e cond (y), given by e cond (y) = φ(y) x 2 p y (x, y) dx Then the mean square error J is given by J = E ( e cond (y) ) because J = φ(y) x 2 p(x, y) dx dy = p y (y) e cond (y) dy

9 0-9 MMSE Estimation S. Lall, Stanford The MMSE estimator We can write the MSE conditioned on y as e cond (y meas ) = E ( φ(y) x 2 y = y meas ) For each y meas, we can pick a value for φ(y meas ) So we have an MMSE prediction problem for each y meas

10 0-0 MMSE Estimation S. Lall, Stanford The MMSE estimator Apply the MVD to z = φ(y) x conditioned on y = w to give e cond (w) = E ( φ(y) x 2 y = w ) = E ( x h(w) 2 y = w ) + φ(w) h(w) 2 where h(w) is the mean of x conditioned on y = w h(w) = E(x y = w) To minimize e cond (w) we therefore pick φ(w) = h(w)

11 0 - MMSE Estimation S. Lall, Stanford The error of the MMSE estimator With this choice of estimator e cond (w) = E ( x h(w) 2 y = w ) = tracecov(x y = w)

12 0-2 MMSE Estimation S. Lall, Stanford Summary: the MMSE estimator The MMSE estimator is φ mmse (y meas ) = E(x y = y meas ) e cond (y meas ) = tracecov(x y = y meas ) We often write ˆx mmse = φ mmse (y meas ) e cond (y meas ) = E ( x ˆx mmse 2 y = ymeas ) The estimate only depends on the conditional pdf of x y = y meas The means and covariance are those of the conditional pdf The above formulae give the MMSE estimate for any pdf on x and y

13 0-3 MMSE Estimation S. Lall, Stanford Example: MMSE estimation (x, y) are uniformly distributed on the triangle A. i.e., the pdf is { 2 if (x, y) A p(x, y) = 0 otherwise the conditional distribution of x y = y meas is uniform on [y meas, ] the MMSE estimator of x given y = y meas is ˆx mmse = E(x y = y meas ) = + y meas 2 the conditional mean square error of this estimate is E ( x ˆx mmse 2 y = ymeas ) = 2 (y meas ) 2

14 0-4 MMSE Estimation S. Lall, Stanford Example: MMSE estimation The mean square error is E ( φ mmse (y) x 2) = E ( e cond (y) ) = x x=0 y=0 p(x, y) e cond (y) dy dx = x=0 x y=0 6 (y )2 dy dx = 24

15 0-5 MMSE Estimation S. Lall, Stanford Example: MMSE estimation (x, y) are uniformly distributed on the L-shaped region A, i.e., the pdf is { 4 p(x, y) = 3 if (x, y) A 0 otherwise the MMSE estimator of x given y = y meas is ˆx mmse = E(x y = y meas ) = { 2 if 0 y meas if 2 < y meas the mean square error of this estimate is E ( x ˆx mmse 2 y = ymeas ) = { 2 if 0 y meas 2 48 if 2 < y meas

16 0-6 MMSE Estimation S. Lall, Stanford Example: MMSE estimation 0.5 The mean square error is E ( φ mmse (y) x 2) = E ( e cond (y) ) = A p(x, y) e cond (y) dy dx = x=0 2 y=0 2 p(x, y) dy dx + x= 2 2 y= 2 48 p(x, y) dy dx = 6

17 0-7 MMSE Estimation S. Lall, Stanford MMSE estimation for Gaussians Suppose [ x y] N(µ, Σ) where µ = [ ] µx µ y Σ = [ ] Σx Σ xy Σ T xy Σ y We know that the conditional density of x y = y meas is N(µ, Σ ) where 3 2 µ = µ x + Σ xy Σ y (y meas µ y ) Σ = Σ x Σ xy Σ y Σ T xy

18 0-8 MMSE Estimation S. Lall, Stanford MMSE estimation for Gaussians The MMSE estimator when x, y are jointly Gaussian is φ mmse (y meas ) = µ x + Σ xy Σ y (y meas µ y ) e cond (y meas ) = trace ( Σ x Σ xy Σ y Σ T xy ) The conditional MSE e cond (y) is independent of y; a special property of Gaussians Hence the optimum MSE achieved is E ( φ mmse (y) x 2) = trace ( ) Σ x Σ xy Σ y Σ T xy The estimate ˆx mmse is an affine function of y meas

19 0-9 MMSE Estimation S. Lall, Stanford Posterior covariance Let s look at the error z = φ(y) x. We have cov(z y = y meas ) = cov ( ) x y = y meas We use this to find a confidence region for the estimate cov(x y = y meas ) is called the posterior covariance

20 0-20 MMSE Estimation S. Lall, Stanford Example: MMSE for Gaussians [ [ ] x Here N(0, Σ) with Σ = y] We measure y meas = and the MMSE estimator is φ mmse (y meas ) = Σ 2 Σ 22 y meas = 0.8y meas = The posterior covariance is Q = Σ x Σ xy Σ y Σ yx = 0.88 Let α 2 = QF (0.9), then α.54 and the confidence interval is χ 2 Prob ( x.6 α ) y = ymeas = 0.9

21 0-2 MMSE Estimation S. Lall, Stanford Bias The MMSE estimator is unbiased; that is the mean error is zero. E ( φ mmse (y) x ) = 0

22 0-22 MMSE Estimation S. Lall, Stanford Estimating a linear function of the unknown Suppose p(x, y) is a pdf on x, y q = Cx is a random variable we measure y and would like to estimate q The optimal estimator is q mmse = C E ( x y = y meas ) Because E ( q y = y meas ) = C E ( x y = ymeas ) The optimal estimate of Cx is C multiplied by the optimal estimate of x Only works for linear functions of x

23 0-23 MMSE Estimation S. Lall, Stanford MMSE and MAP estimation Suppose x and y are random variables, with joint pdf f(x, y) The maximum a posterior (MAP) estimate is the x that maximizes h(x, y meas ) = conditional pdf of x (y = y meas ) The MAP estimate also maximizes the joint pdf x map = arg max x f(x, y meas ) 3 2 When x, y are jointly Gaussian, then the peak of the conditional pdf is the conditional mean. 0 i.e., for Gaussians, the MMSE estimate is equal to the MAP estimate