Projection/Rejection Matrices and Projected/Rejected Vectors

1. The concepts of Projection / Rejection Matrices and Projection / Rejection Vectors apply to only Basis Vector Matrices.
2. Any Vector that belongs to the Column Space of a Basis Vector Matrix is a Projected Vector on that Basis Vector Matrix. A Projected Vector is also called a Projection ON the Vector Space/Subspace represented by the Basis Vector Matrix.
3. A Rejected Vector is the Difference between Any Vector and its Corresponding Projected Vector on the Basis Vector Matrix. A Rejected Vector is also called a Rejection FROM the Vector Space/Subspace represented by the Basis Vector Matrix.
4. Given Any \(M \times N\) Basis Vector Matrix \(A\), the Projection Matrix \(P\) is any corresponding \(M\times M\) Matrix which when Multiplied to Any \(M\times 1\) Vector \(B\) gives a Resultant \(M\times 1\) Vector \(X\) which belongs to the Column Space of the Basis Vector Matrix \(A\) i.e. the Matrix \(P\) Projects the Vector \(B\) ON the Basis Vector Matrix \(A\) giving the Projected Vector / Projection \(X\). That is
   
   \(PB=X\)   ...(1)
   
   Please note that if the Vector \(B\) is already in the Column Space of Basis Vector Matrix \(A\) then Multiplying it with the Projection Matrix \(P\) gives back the Same Vector \(B\).
   
   Also note that if the Vector \(B\) is in the Orthogonal Space of Basis Vector Matrix \(A\) then Multiplying it with the Projection Matrix \(P\) gives back a NULL Vector.
5. Given Any \(M \times N\) Basis Vector Matrix \(A\) and corresponding \(M\times M\) Projection Matrix \(P\), the Rejection Matrix \(R\) is any corresponding \(M\times M\) Matrix which when Multiplied to Any \(M\times 1\) Vector \(B\) gives a Resultant \(M\times 1\) Vector \(Y\) which is a Difference between Vector \(B\) and the Corresponding Projected Vector \(X\) i.e. the Matrix \(R\) Rejects the Vector \(B\) FROM the Basis Vector Matrix \(A\) giving the Rejected Vector / Rejection \(Y\). That is
   
   \(RB=Y=B-X\)   ...(2)
   
   Please note that if the Vector \(B\) is already in the Column Space of Basis Vector Matrix \(A\) then Multiplying it with the Rejection Matrix \(R\) gives back a NULL Vector.
   
   Also note that if the Vector \(B\) is in the Orthogonal Space of Basis Vector Matrix \(A\) then Multiplying it with the Rejection Matrix \(R\) gives back the Same Vector \(B\).
6. From equation (2) we can see that Any Vector can be given as a Sum of its Projection ON and Rejection FROM any given Basis Vector Matrix \(A\). That is since
   
   \(Y=B-X\hspace{.5cm}\Rightarrow B=X+Y\)   ...(3)
   
   Putting the Value of \(X\) and \(Y\) from equations (1) and (2) we have
   
   \(B=PB+RB\hspace{.5cm}\Rightarrow B=(P+R)B=IB\)   ...(4)
   
   Thus, as given is equation (4) above, the Sum of Projection and Corresponding Rejection Matrix is an Identity Matrix. This implies
   
   \(R=I-P\)   ...(5)
7. The Projection Matrix corresponding to a Vector Space represented by a \(N \times N\) Square Basis Vector Matrix is a \(N \times N\) Identity Matrix. The Rejection Matrix corresponding to a Vector Space represented by a \(N \times N\) Square Basis Vector Matrix is a \(N \times N\) NULL Matrix. This is because every \(N\)-Dimensional Vector is in the Column Space of any \(N \times N\) Basis Vector Matrix.
8. The following gives the Derivation for Formula of Projection Matrix \(P\) and corresponding Rejection Matrix \(R\) for a given \(M \times N\) Basis Vector Matrix \(A\)
   
   Let \(X\) be any Vector which May or May Not Belong to the Column Space of \(A\). This implies \(AX\) is a Vector that Belongs to Column Space of \(A\).
   
   Let \(B\) be any Vector which Does Not Belong to the Column Space of \(A\) and Let \(AX\) be the Projection of Vector \(B\) ON Basis Vector Matrix \(A\). That is
   
   \(PB=AX\)   ...(6)
   
   This implies the \(B-AX\) is the Rejection Vector from Basis Vector Matrix \(A\).
   
   Now, Let \(C\) be another \(M \times N\) Basis Vector Matrix that is Orthogonal to the Rejection Vector \(B-AX\). This implies
   
   \(C^T(B-AX)=0\)
   
   \(\Rightarrow C^TB-C^TAX=0\)
   
   \(\Rightarrow C^TAX=C^TB\)
   
   \(\Rightarrow X={(C^TA)}^{-1}C^TB\)   ...(7)
   
   Multiplying equation (7) with Basis Vector Matrix \(A\) on Both Sides we get
   
   \(AX=A{(C^TA)}^{-1}C^TB\)   ...(8)
   
   Now, From equations (6) and (8) we get
   
   \(PB=A{(C^TA)}^{-1}C^TB\)
   
   \(\Rightarrow P=A{(C^TA)}^{-1}C^T\)   ...(9)
   
   If Basis Vector Matrix \(A\) consists of Vectors \(\mathbf{a_1}, \mathbf{a_2}, ..., \mathbf{a_n}\) and Basis Vector Matrix \(C\) consists of Vectors \(\mathbf{c_1}, \mathbf{c_2}, ..., \mathbf{c_n}\) then equation (9) can be given as
   
   \(P=\begin{bmatrix}a_1 & a_2 & \cdots & \ a_n\end{bmatrix} {\begin{bmatrix}{c_1}^Ta_1 & {c_1}^Ta_2 & \cdots & \ {c_1}^Ta_n\\ {c_2}^Ta_1 & {c_2}^Ta_2 & \cdots & \ {c_2}^Ta_n \\ \vdots & \vdots & \ddots & \vdots \\ {c_n}^Ta_1 & {c_n}^Ta_2 & \cdots & \ {c_n}^Ta_n \end{bmatrix}}^{-1}\begin{bmatrix}{c_1}^T \\ {c_2}^T \\ \vdots \\ {c_n}^T\end{bmatrix} \)   ...(10)
   
   The Rejection Matrix can be calculated by putting the value of Projection Matrix from equation (9) in equation (5) as follows
   
   \(R=I-P\hspace{.5cm}\Rightarrow R=I-A{(C^TA)}^{-1}C^T\)   ...(11)
   
   The equations (9) and (11) give the formula for Projection and Rejection Matrices respectively.
   
   Now, if Basis Vector Matrix \(C=A\) then the Projection Matrix obtained is an Orthogonal Projection Matrix and is given by replacing \(C\) with \(A\) in equation (9) as
   
   \(P=A{(A^TA)}^{-1}A^T\)   ...(12)
   
   If Basis Vector Matrix \(A\) consists of Vectors \(\mathbf{a_1}, \mathbf{a_2}, ..., \mathbf{a_n}\) then equation (12) can be given as
   
   \(\Rightarrow P=\begin{bmatrix}a_1 & a_2 & \cdots & \ a_n\end{bmatrix} {\begin{bmatrix}{a_1}^Ta_1 & {a_1}^Ta_2 & \cdots & \ {a_1}^Ta_n\\ {a_2}^Ta_1 & {a_2}^Ta_2 & \cdots & \ {a_2}^Ta_n \\ \vdots & \vdots & \ddots & \vdots \\ {a_n}^Ta_1 & {a_n}^Ta_2 & \cdots & \ {a_n}^Ta_n \end{bmatrix}}^{-1}\begin{bmatrix}{a_1}^T \\ {a_2}^T \\ \vdots \\ {a_n}^T\end{bmatrix} \)   ...(13)
   
   Also, if Basis Vector Matrix \(C=A\) then the Rejection Matrix obtained is an Orthogonal Rejection Matrix and is given by replacing \(C\) with \(A\) in equation (11) as
   
   \(R=I-P=I-A{(A^TA)}^{-1}A^T\)   ...(14)
   
   The equations (12) and (14) give the formula for Orthogonal Projection and Rejection Matrices respectively. The Projection and Rejection Vectors Obtained by Multiplying these Matrices with any Vector are Orthogonal to each other. Also the Rejection Vector thus obtained is Orthogonal to the Basis Vector Matrix \(A\).
   
   Now, If the Vectors \(\mathbf{a_1}, \mathbf{a_2}, ..., \mathbf{a_n}\) of Basis Vector Matrix \(A\) are Mutually Orthogonal then equation (13) becomes
   
   \(P=\begin{bmatrix}a_1 & a_2 & \cdots & \ a_n\end{bmatrix} {\begin{bmatrix}{a_1}^Ta_1 & 0 & \cdots & \ 0\\ 0& {a_2}^Ta_2 & \cdots & \ 0\\ \vdots & \vdots & \ddots & \vdots \\ 0 & 0& \cdots & \ {a_n}^Ta_n \end{bmatrix}}^{-1}\begin{bmatrix}{a_1}^T \\ {a_2}^T \\ \vdots \\ {a_n}^T\end{bmatrix} \)   ...(15)
   
   \(\Rightarrow P=\begin{bmatrix}a_1 & a_2 & \cdots & \ a_n\end{bmatrix} \begin{bmatrix}\frac{1}{{a_1}^Ta_1} & 0 & \cdots & \ 0\\ 0& \frac{1}{{a_2}^Ta_2} & \cdots & \ 0\\ \vdots & \vdots & \ddots & \vdots \\ 0 & 0& \cdots & \ \frac{1}{{a_n}^Ta_n} \end{bmatrix}\begin{bmatrix}{a_1}^T \\ {a_2}^T \\ \vdots \\ {a_n}^T\end{bmatrix} \)   ...(16)
   
   \(\Rightarrow P=\frac{a_1{a_1}^T}{{a_1}^Ta_1} + \frac{a_2{a_2}^T}{{a_2}^Ta_2} + \cdots + \ \frac{a_n{a_n}^T}{{a_n}^Ta_n}\)   ...(17)
   
   \(\Rightarrow P=P_1 + P_2 + \cdots + P_n\)   ...(18)
   
   where \(P_1=\frac{a_1{a_1}^T}{{a_1}^Ta_1}\) (Projection Matrix for Vector \(\mathbf{a_1}\)), \(P_2=\frac{a_2{a_2}^T}{{a_2}^Ta_2}\) (Projection Matrix for Vector \(\mathbf{a_2}\)), ... , \(P_n=\frac{a_n{a_n}^T}{{a_n}^Ta_n}\) (Projection Matrix for Vector \(\mathbf{a_n}\))
   
   So, equations (17) and (18) state that Orthogonal Projection Matrix for any Basis Vector Matrix having Mutually Orthogonal Vectors is Sum of Projection Matrix of Each Vector.
   
   Correspondingly for Basis Vector Matrix \(A\) having Mutually Orthogonal Vectors, the Orthogonal Rejection Matrix is given as
   
   \(R= I- (\frac{a_1{a_1}^T}{{a_1}^Ta_1} + \frac{a_2{a_2}^T}{{a_2}^Ta_2} + \cdots + \ \frac{a_n{a_n}^T}{{a_n}^Ta_n})\)   ...(19)
   
   \(\Rightarrow R=I-(P_1 + P_2 + \cdots + P_n)\)   ...(20)
   
   Following are some Additional Properties of Orthogonal Projection and Rejection Matrices for Basis Vector Matrices having Mutually Orthogonal Vectors
   1. The Product of Projection Matrices for any 2 or More of the Vectors of the Vectors Space (In Any Order) is a NULL Matrix. That is, if \(P_1, P_2, ..., P_n\) are Projection Matrices corresponding the Vectors \(\mathbf{a_1}, \mathbf{a_2}, ..., \mathbf{a_n}\) of Basis Vector Matrix \(A\) then the Matrix Products \(P_1P_2\), \(P_2P_1\), \(P_2P_3\), \(P_2P_1P_3, P_1P_2\cdots P_n\) etc. are all equal to NULL Matrix.
   2. The Rejection Matrix of the Vectors Space is the Product of Rejection Matrices of All the Vectors (In Any Order). That is, if \(R_1, R_2, ..., R_n\) are Rejection Matrices corresponding the Vectors \(\mathbf{a_1}, \mathbf{a_2}, ..., \mathbf{a_n}\) of Basis Vector Matrix \(A\) then the Rejection Matrix \(R\) of Basis Vector Matrix \(A\) can be given as
      
      \(R=R_1R_2...R_N\)
      
      \(\Rightarrow R=(I-P_1)(I-P_2)\cdots(I-P_n)\)
      
      \(\Rightarrow R=I-(P_1 + P_2 + \cdots + P_n)\)   (\(\because\) Matrix Products \(P_1P_2\), \(P_2P_1\), \(P_2P_3\), \(P_2P_1P_3, P_1P_2\cdots P_n\) etc. are all equal to NULL Matrix)
9. All Projection and Rejection Matrices are Indempotent.