Subject Exercise A We consider n observations y1, . . . , yn of a variable and n vectors xi (t(xi) = (xi1, . . . , xik)) where we have the observations of k variables. For i ∈ {1, . . . , n}, we...

Exercise A
1.
The likelihood function is the density function regarded as a function of θ.
L(θ|x) = f(x|θ), θ ∈ Θ.
The maximum likelihood estimator (MLE),
ˆθ(x) = arg max θ L(θ|x).
2.
A random sample x1, x2,...,xn from a distribution f(x) is a set of independently and identically variables with xi ∼ f(x) for all i.
Their joint p.d.f is
f(x1, x2,...,xn) = f(x1)f(x2)··· f(x2) = n i=1 f(xi). The sample moments provide estimates of the moments of f(x). We need to know how they are distributed.
The mean ¯x of a random sample is an unbiased estimate of the population moment µ = E(x), since E(¯x) = E xin = 1n E(xi) = nnµ = µ.
The variance of a sum of independent variables is the sum of their variances, since covariances are zero.
Therefore V (¯x) = V xin = 1n2 V (xi) = nn2 σ2 = σ2n . Observe that V (¯x) → 0 as n → ∞. Since E(¯x) = µ, the estimates become increasingly concentrated around the true population parameter.
Such an estimate is said to be consistent. The sample variance is not an unbiased estimate of σ 2 = V (x),
since E(s 2) = E 1 n (xi − x¯)
2 = E 1 n (xi − µ)+( µ − x¯)
2 = E 1 n (xi − µ) 2 + 2( xi − µ)( µ − x¯)+( µ − x¯) 2 = V (x) − 2 E {(¯x − µ) 2 } + E {(¯x − µ) 2 } = V (x) − V (¯x).
Here, we have used the result that E 1 n (xi − µ)( µ − x¯) = − E {(µ − x¯) 2 } = − V (¯x). It follows that E(s 2) = V (x) − V (¯x) = σ 2 − σ 2 n = σ 2 (n − 1) n . Therefore, s 2 is a biased estimator of the population variance. For an unbiased estimate, we should use σˆ 2 = s 2 n n − 1 =
(xi − x¯)2 n − 1 . However, s 2 is still a consistent estimator, since E(s 2) → σ 2 as n → ∞ and also V (s 2) → 0. The value of V (s 2) depends on the distribution of underlying population, which is often assumed to be a normal.
3.
Let B be the set of all possible vectors . If there is no further information, the B is k -dimensional real Euclidean space.
The object is to find a vector 1 2 ' ( , ,..., ) k b bb b from B that minimizes the sum of squared deviations of ' ,
i s i.e., 2 1 ( ) ' ( )'( ) n i i S y X y X for given y and X.
A minimum will always exist as S( ) is a real-valued, convex and differentiable function. Write S() ' ' ' 2' ' y y XX X y . Differentiate S( ) with respect to 2 2 ( ) 2' 2' ( ) 2 ' (atleast non-negative definite). S XX Xy S X X
The normal equation is ( ) 0 ' ' S X Xb X y where the following result is used: Result: If f () ' z Z AZ is a quadratic form, Z is a m1 vector and A is any m m symmetric matrix then F z Az () 2 z .
Since it is assumed that rank ( ) X k (full rank), then X ' X is a positive definite and unique solution of the normal equation is 1 b XX Xy (') ' which is termed as ordinary least squares estimator (OLSE) of . Since 2 2 S( ) is at least non-negative definite, so b minimize S( ) .
4.
To show that Φ is an isomorphism, it suffices to show that Φ is linear and that Φ has an inverse function.
To see that Φ is linear, if A1, A2 ∈ Mn×n(F) and c1, c2 ∈ F are arbitrary, then Φ(c1A1 + c2A2) = B −1 (c1A1 + c2A2)B = (B −1 (c1A1) + B −1 (c2A2))B = (c1B −1A1 + c2B −1A2)B = c1B −1A1B + c2B −1A2B = c1Φ(A1) + c2Φ(A2). To see that Φ has an inverse, define Ψ : Mn×n(F) → Mn×n(F) by Ψ(A) = BAB−1 .
Then (Φ ◦ Ψ)(A) = Φ(Ψ(A)) = Φ(BAB−1 ) = B −1 (BAB−1 )B = A; (Ψ ◦ Φ)(A) = Ψ(Φ(A)) = Ψ(B −1AB) = B(B −1AB)B −1 = A,
5.
The centered Gaussian random variable X on R with variance σ 2 > 0 has density given by p(x) = 1 √ 2πσ2 exp −x 2 2σ 2 .
It plays a central role in statistics due to the central limit theorem. It also has a central position in statistical and signal processing estimation problems. Important properties of this...