Chapter 7 Bilinear and quadratic forms

A bilinear form on $\mathbb{R}^{n}$ is a function $L=L(\mathbf{x},\mathbf{y})$ of two arguments $\mathbf{x},\mathbf{y}\in\mathbb{R}^{n}$ which is linear in each argument, i.e. such that

1. 1.

   $L(\alpha\mathbf{x}_{1}+\beta\mathbf{x}_{2},\mathbf{y})=\alpha L(\mathbf{x}_{1},\mathbf{y})+\beta L(\mathbf{x}_{2},\mathbf{y});$

2. 2.

   $L(\mathbf{x},\alpha\mathbf{y}_{1}+\beta\mathbf{y}_{2})=\alpha L(\mathbf{x},\mathbf{y}_{1})+\beta L(\mathbf{x},\mathbf{y}_{2})$.

One can consider bilinear form whose values belong to an arbitrary vector space, but in this book we only consider forms that take real values.

If $\mathbf{x}=(x_{1},x_{2},\ldots,x_{n})^{T}$ and $\mathbf{y}=(y_{1},y_{2},\ldots,y_{n})^{T}$, a bilinear form can be written as

or in matrix form

where

The matrix $A$ is determined uniquely by the bilinear form $L$.

There are several equivalent definition of a quadratic form.

One can say that a quadratic form on $\mathbb{R}^{n}$ is the “diagonal” of a bilinear form $L$, i.e. that any quadratic form $Q$ is defined by $Q[\mathbf{x}]=L(\mathbf{x},\mathbf{x})=(A\mathbf{x},\mathbf{x})$.

Another, more algebraic way, is to say that a quadratic form is a homogeneous polynomial of degree 2, i.e. that $Q[\mathbf{x}]$ is a polynomial of $n$ variables $x_{1},x_{2},\ldots,x_{n}$ having only terms of degree 2. That means that only terms $ax_{k}^{2}$ and $cx_{j}x_{k}$ are allowed.

There many ways (in fact, infinitely many) to write a quadratic form $Q[\mathbf{x}]$ as $Q[\mathbf{x}]=(A\mathbf{x},\mathbf{x})$. For example, the quadratic form $Q[\mathbf{x}]=x_{1}^{2}+x_{2}^{2}-4x_{1}x_{2}$ on $\mathbb{R}^{2}$ can be represented as $(A\mathbf{x},\mathbf{x})$ where $A$ can be any of the matrices

In fact, any matrix $A$ of form

will work.

But if we require the matrix $A$ to be symmetric, then such a matrix is unique:

Any quadratic form $Q[\mathbf{x}]$ on $\mathbb{R}^{n}$ admits unique representation $Q[\mathbf{x}]=(A\mathbf{x},\mathbf{x})$ where $A$ is a (real) symmetric matrix.

For example, for the quadratic form

on $\mathbb{R}^{3}$, the corresponding symmetric matrix $A$ is

One can also define a quadratic form on $\mathbb{C}^{n}$ (or any complex inner product space) by taking a self-adjoint transformation $A=A^{*}$ and defining $Q$ by $Q[\mathbf{x}]=(A\mathbf{x},\mathbf{x})$. While our main examples will be in $\mathbb{R}^{n}$, all the theorems are true in the setting of $\mathbb{C}^{n}$ as well. Bearing this in mind, we will always use $A^{*}$ instead of $A^{T}$

The only essential difference with the real case is that in the complex case we do not have any freedom of choice: if the quadratic form is real, the corresponding matrix has to be Hermitian (self-adjoint).

Note that if $A=A^{*}$ then

so $(A\mathbf{x},\mathbf{x})\in\mathbb{R}$.

The converse is also true.

We leave the proof as an exercise for the reader, see Problem 7.1.4 below.

One of the possible ways to prove Lemma 7.1.1 is to use the following version of polarization identities.

For the proof of Lemma 7.1.2 see Exercise 5.6.3 in Chapter 5 above.

Let us have a quadratic form $Q[\mathbf{x}]=(A\mathbf{x},\mathbf{x})$ in $\mathbb{F}^{n}$ ($\mathbb{F}$ is $\mathbb{R}$ or $\mathbb{C}$). Introduce new variables $\mathbf{y}=(y_{1},y_{2},\ldots,y_{n})^{T}\in\mathbb{F}^{n}$, with $\mathbf{y}=S^{-1}\mathbf{x}$, where $S$ is some invertible $n\times n$ matrix, so $\mathbf{x}=S\mathbf{y}$.

Then,

so in the new variables $\mathbf{y}$ the quadratic form has matrix $S^{*}AS$.

So, we want to find an invertible matrix $S$ such that the matrix $S^{*}AS$ is diagonal. Note, that it is different from the diagonalization of matrices we had before: we tried to represent a matrix $A$ as $A=SDS^{-1}$, so the matrix $D=S^{-1}AS$ was diagonal. However, for unitary matrices $U$, we have $U^{*}=U^{-1}$, and we can orthogonally diagonalize symmetric matrices. So we can apply the orthogonal diagonalization we studied before to the quadratic forms.

Namely, we can represent the matrix $A$ as $A=UDU^{*}=UDU^{-1}$. Recall, that $D$ is a diagonal matrix with eigenvalues of $A$ on the diagonal, and $U$ is the matrix of eigenvectors (we need to pick an orthogonal basis of eigenvectors). Then $D=U^{*}AU$, so in the variables $\mathbf{y}=U^{-1}\mathbf{x}$ the quadratic form has diagonal matrix.

Let us analyze the geometric meaning of the orthogonal diagonalization. The columns $\mathbf{u}_{1},\mathbf{u}_{2},\ldots,\mathbf{u}_{n}$ of the unitary matrix $U$ form an orthonormal basis in $\mathbb{F}^{n}$, let us call this basis $\mathcal{B}$. The change of coordinate matrix $[I]_{{}_{\scriptstyle\mathcal{S},\mathcal{B}}}$ from this basis to the standard one is exactly $U$. We know that $\mathbf{y}=(y_{1},y_{2},\ldots,y_{n})^{T}=U^{-1}\mathbf{x}$, so the coordinates $y_{1},y_{2},\ldots,y_{n}$ can be interpreted as coordinates of the vector $\mathbf{x}$ in the new basis $\mathbf{u}_{1},\mathbf{u}_{2},\ldots,\mathbf{u}_{n}$.

So, orthogonal diagonalization allows us to visualize very well the set $Q[\mathbf{x}]=1$, or a similar one, as long as we can visualize it for diagonal matrices.

Orthogonal diagonalization involves computing eigenvalues and eigenvectors, so it may be difficult to do without computers for large $n$. On the other hand, the non-orthogonal diagonalization, i.e. finding an invertible $S$ (without requiring $S^{-1}=S^{*}$) such that $D=S^{*}AS$ is diagonal, is much easier computationally and can be done using only algebraic operations (addition, subtraction, multiplication, division).

Below we present two most used methods of non-orthogonal diagonalization.

It is an easy exercise to see that a $2\times 2$ matrix

is positive definite if and only if

Indeed, if $a>0$ and $\det A=ac-|b|^{2}>0$, then $c>0$, so $\operatorname{trace}A=a+c>0$. So we know that if $\lambda_{1},\lambda_{2}$ are eigenvalues of $A$ then $\lambda_{1}\lambda_{2}>0$ ($\det A>0$) and $\lambda_{1}+\lambda_{2}=\operatorname{trace}A>0$. But that only possible if both eigenvalues are positive. So we have proved that conditions (7.4.1) imply that $A$ is positive definite. The opposite implication is quite simple, we leave it as an exercise for the reader.

This result can be generalized to the case of $n\times n$ matrices. Namely, for a matrix $A$

let us consider its all upper left submatrices

First of all let us notice that if $A>0$ then $A_{k}>0$ also (can you explain why?). Therefore, since all eigenvalues of a positive definite matrix are positive, see Theorem 7.4.1, $\det A_{k}>0$ for all $k$.

One can show that if $\det A_{k}>0$ $\forall k$ then all eigenvalues of $A$ are positive by analyzing diagonalization of a quadratic form using row and column operations, which was described in Section 7.2.2. The key here is the observation that if we perform row/column operations in natural order (i.e. first subtracting the first row/column from all other rows/columns, then subtracting the second row/columns from the rows/columns $3,4,\ldots,n$, and so on…), and if we are not doing any row interchanges, then we automatically diagonalize quadratic forms $A_{k}$ as well. Namely, after we subtract first and second rows and columns, we get diagonalization of $A_{2}$; after we subtract the third row/column we get the diagonalization of $A_{3}$, and so on.

Since we are performing only row replacement we do not change the determinant. Moreover, since we are not performing row exchanges and performing the operations in the correct order, we preserve determinants of $A_{k}$. Therefore, the condition $\det A_{k}>0$ guarantees that each new entry in the diagonal is positive.

Of course, one has to be sure that we can use only row replacements, and perform the operations in the correct order, i.e. that we do not encounter any pathological situation. If one analyzes the algorithm, one can see that the only bad situation that can happen is the situation where at some step we have zero in the pivot place. In other words, if after we subtracted the first $k$ rows and columns and obtained a diagonalization of $A_{k}$, the entry in the $k+1$st row and $k+1$st column is $0$. We leave it as an exercise for the reader to show that this is impossible. ∎

The proof we outlined above is quite simple. However, let us present, in more detail, another one, which can be found in more advanced textbooks. I personally prefer this second proof, for it demonstrates some important connections.

We will need the following characterization of eigenvalues of a hermitian matrix.

For a subspace $E$ of an inner product space $X$ the codimension $\operatorname{codim}E$ of $E$ is defined as the the dimension of its orthogonal complement, $\operatorname{codim}E=\dim(E^{\perp})$. Since for a subspace $E\subset X$, $\dim X=n$ we have $\dim E+\dim E^{\perp}=n$, we can see that $\operatorname{codim}E=\dim X-\dim E$.

Recall that the trivial subspace $\{\mathbf{0}\}$ has dimension zero, so the whole space $X$ has codimension $0$.

Let us explain in more details what the expressions like $\max\min$ and $\min\max$ mean. To compute the first one, we need to consider all subspaces $E$ of dimension $k$. For each such subspace $E$ we consider the set of all $\mathbf{x}\in E$ of norm 1, and find the minimum of $(A\mathbf{x},\mathbf{x})$ over all such $\mathbf{x}$. Thus for each subspace we obtain a number, and we need to pick a subspace $E$ such that the number is maximal. That is the $\max\min$.

The $\min\max$ is defined similarly.