Gradient Vector, Jacobian Matrix and Hessian Matrix

1. Gradient or Gradient Vector of any Multi-Variable Function is the Array/List of all Partial Derivatives of the Function with respect to all its Variables. For example, given a Function \(F\) of \(N\) Variables \(x_1, x_2, x_3, \cdots, x_n\), the Gradient of Function \(F\) is given as
   
   \(grad\hspace{2mm}F = \begin{bmatrix}{\Large \frac{\partial F}{\partial x_1}} & {\Large \frac{\partial F}{\partial x_2}} & {\Large \frac{\partial F}{\partial x_3}} & \cdots & {\Large \frac{\partial F}{\partial x_n}}\end{bmatrix}\)
2. A Jacobian Matrix is an Array/List of Gradient of Functions of Mutiple Variables. Given a set of \(K\) Multi-Variable Functions \(F_1, F_2, F_3, \cdots, F_k\) dependent on same set of \(N\) Variables \(x_1, x_2, x_3, \cdots, x_n\), the Jacobian Matrix \(J\) is given by the Partial Derivatives of the Functions as follows
   
   \(J = \begin{bmatrix}{\Large \frac{\partial F_1}{\partial x_1}} & {\Large \frac{\partial F_1}{\partial x_2}} & {\Large \frac{\partial F_1}{\partial x_3}} & \cdots & {\Large \frac{\partial F_1}{\partial x_n}} \\ {\Large \frac{\partial F_2}{\partial x_1}} & {\Large \frac{\partial F_2}{\partial x_2}} & {\Large \frac{\partial F_2}{\partial x_3}} & \cdots & {\Large \frac{\partial F_2}{\partial x_n}} \\ {\Large \frac{\partial F_3}{\partial x_1}} & {\Large \frac{\partial F_3}{\partial x_2}} & {\Large \frac{\partial F_3}{\partial x_3}} & \cdots & {\Large \frac{\partial F_3}{\partial x_n}} \\ \vdots & \vdots & \vdots & \ddots & \vdots \\ {\Large \frac{\partial F_k}{\partial x_1}} & {\Large \frac{\partial F_k}{\partial x_2}} & {\Large \frac{\partial F_k}{\partial x_3}} & \cdots & {\Large \frac{\partial F_k}{\partial x_n}} \end{bmatrix}\)
3. A Hessian Matrix is a Square Matrix of Second-Order Partial Derivatives of a Multi-Variable Scalar Function. For example, given a Function \(F\) of \(N\) Variables \(x_1, x_2, x_3, \cdots, x_n\), the \(N \times N\) Hessian Matrix for the function is given as
   
   \(H_F = \begin{bmatrix}{\Large \frac{{\partial}^2 F}{{\partial x_1}^2}} & {\Large \frac{{\partial}^2 F}{\partial x_1\partial x_2}} & {\Large \frac{{\partial}^2 F}{\partial x_1\partial x_3}} & \cdots & {\Large \frac{{\partial}^2 F}{\partial x_1\partial x_n}} \\ {\Large \frac{{\partial}^2 F}{\partial x_2\partial x_1}} & {\Large \frac{{\partial}^2 F}{{\partial x_2}^2}} & {\Large \frac{{\partial}^2 F}{\partial x_2\partial x_3}} & \cdots & {\Large \frac{{\partial}^2 F}{\partial x_2\partial x_n}} \\ {\Large \frac{{\partial}^2 F}{\partial x_3\partial x_1}} & {\Large \frac{{\partial}^2 F}{\partial x_3\partial x_2}} & {\Large \frac{{\partial}^2 F}{{\partial x_3}^2}} & \cdots & {\Large \frac{{\partial}^2 F}{\partial x_3\partial x_n}} \\ \vdots & \vdots & \vdots & \ddots & \vdots \\ {\Large \frac{{\partial}^2 F}{\partial x_n\partial x_1}} & {\Large \frac{{\partial}^2 F}{\partial x_n\partial x_2}} & {\Large \frac{{\partial}^2 F}{\partial x_n\partial x_3}} & \cdots & {\Large \frac{{\partial}^2 F}{{\partial x_n}^2}} \end{bmatrix}\)