Chapter 4 Introduction to spectral theory (eigenvalues and eigenvectors)

A scalar $\lambda$ is called an eigenvalue of an operator $A:V\to V$ if there exists a non-zero vector $\mathbf{v}\in V$ such that

$$A\mathbf{v}=\lambda\mathbf{v}.$$

The vector $\mathbf{v}$ is called the eigenvector of $A$ (corresponding to the eigenvalue $\lambda$).

If we know that $\lambda$ is an eigenvalue, the eigenvectors are easy to find: one just has to solve the equation $A\mathbf{x}=\lambda\mathbf{x}$, or, equivalently

$$(A-\lambda I)\mathbf{x}=\mathbf{0}.$$

So, finding all eigenvectors, corresponding to an eigenvalue $\lambda$ is simply finding the nullspace of $A-\lambda I$. The nullspace $\operatorname{Ker}(A-\lambda I)$, i.e. the set of all eigenvectors and $\mathbf{0}$ vector, is called the eigenspace.

The set of all eigenvalues of an operator $A$ is called spectrum of $A$, and is usually denoted $\sigma(A)$.

A scalar $\lambda$ is an eigenvalue if and only if the nullspace $\operatorname{Ker}(A-\lambda I)$ is non-trivial (so the equation $(A-\lambda I)\mathbf{x}=\mathbf{0}$ has a non-trivial solution).

Let $A$ act on $\mathbb{F}^{n}$ (i.e. $A:\mathbb{F}^{n}\to\mathbb{F}^{n}$). Since the matrix of $A$ is square, $A-\lambda I$ has a non-trivial nullspace if and only if it is not invertible. We know that a square matrix is not invertible if and only if its determinant is $0$. Therefore

$$\framebox{$\displaystyle\lambda\in\sigma(A),\text{ i.e.~{}$\lambda$ is an eigenvalue of $A$}\quad\Longleftrightarrow\quad\det(A-\lambda I)=0$}$$

If $A$ is an $n\times n$ matrix, the determinant $\det(A-\lambda I)$ is a polynomial of degree $n$ of the variable $\lambda$. This polynomial is called the characteristic polynomial of $A$. So, to find all eigenvalues of $A$ one just needs to compute the characteristic polynomial and find all its roots.

This method of finding the spectrum of an operator is not very practical in higher dimensions. Finding roots of a polynomial of high degree can be a very difficult problem, and it is impossible to solve the equation of degree higher than 4 in radicals. So, in higher dimensions different numerical methods of finding eigenvalues and eigenvectors are used.

So we know how to find the spectrum of a matrix. But how do we find eigenvalues of an operator acting in an abstract vector space? The recipe is simple:

Take an arbitrary basis, and compute eigenvalues of the matrix of the operator in this basis.

But how do we know that the result does not depend on a choice of the basis?

There can be several possible explanations. One is based on the notion of similar matrices. Let us recall that square matrices $A$ and $B$ are called similar if there exist an invertible matrix $S$ such that

$$A=SBS^{-1}.$$

Note, that determinants of similar matrices coincide. Indeed

$$\det A=\det(SBS^{-1})=\det S\det B\det S^{-1}=\det B$$

because $\det S^{-1}=1/\det S$. Note that if $A=SBS^{-1}$ then

$$A-\lambda I=SBS^{-1}-\lambda SIS^{-1}=S(BS^{-1}-\lambda IS^{-1})=S(B-\lambda I)S^{-1},$$

so the matrices $A-\lambda I$ and $B-\lambda I$ are similar. Therefore

$$\det(A-\lambda I)=\det(B-\lambda I),$$

i.e.

characteristic polynomials of similar matrices coincide.

If $T:V\to V$ is a linear transformation, and $\mathcal{A}$ and $\mathcal{B}$ are two bases in $V$, then

$$[T]_{{}_{\scriptstyle\mathcal{A}\mathcal{A}}}=[I]_{{}_{\scriptstyle\mathcal{A}\mathcal{B}}}[T]_{{}_{\scriptstyle\mathcal{B}\mathcal{B}}}[I]_{{}_{\scriptstyle\mathcal{B}\mathcal{A}}}$$

and since $[I]_{{}_{\scriptstyle\mathcal{B}\mathcal{A}}}=([I]_{{}_{\scriptstyle\mathcal{A}\mathcal{B}}})^{-1}$ the matrices $[T]_{{}_{\scriptstyle\mathcal{A}\mathcal{A}}}$ and $[T]_{{}_{\scriptstyle\mathcal{B}\mathcal{B}}}$ are similar.

In other words, matrices of a linear transformation in different bases are similar.

Therefore, we can define the characteristic polynomial of an operator as the characteristic polynomial of its matrix in some basis. As we have discussed above, the result does not depend on the choice of the basis, so characteristic polynomial of an operator is well defined.

The fundamental theorem of algebra asserts that any polynomial (of degree at least $1$) has a complex root. That implies that an operator in a finite-dimensional complex vector space has at least one eigenvalue, so its spectrum is non-empty.

On the other hand it is easy to construct a linear transformation in a real vector space without real eigenvalues, the rotation $R_{\alpha}$, $\alpha\neq\pi n$ in $\mathbb{R}^{2}$ being one of examples. Since it is usually assumed that eigenvalues should belong to the field of scalars (if an operator acts in a vector space over a field $\mathbb{F}$ the eigenvalues should be in $\mathbb{F}$), such operators have empty spectrum.

Thus, the complex case (i.e. operators acting in complex vector spaces) seems to be the most natural setting for the spectral theory. Since $\mathbb{R}\subset\mathbb{C}$, we can always treat a real $n\times n$ matrix as an operator in $\mathbb{C}^{n}$ to allow complex eigenvalues. Treating real matrices as operators in $\mathbb{C}^{n}$ is typical in the spectral theory, and we will follow this agreement. Finding eigenvalues of a matrix (unless otherwise specified) will always mean finding all complex eigenvalues and not restricting oneself only to real ones.

Note that an operator in an abstract real vector space also can be interpreted as an operator in a complex space. A naïve approach would be to fix a basis (recall that all spaces in this chapter are finite-dimensional), and then work with coordinates in this basis allowing complex coordinates: that will be essentially move from operators in $\mathbb{R}^{n}$ to operators $\mathbb{C}^{n}$ described above.

This construction describes what is known as the complexification of a real vector space, and the result does not depend on the choice of a basis. A “high brow” abstract construction of the complexification, explaining why the result does not depend on the choice of a basis is described below in Section 5.8.2 of Chapter 5.

Let us remind the reader, that if $p$ is a polynomial, and $\lambda$ is its root (i.e. $p(\lambda)=0$) then $z-\lambda$ divides $p(z)$, i.e. $p$ can be represented as $p(z)=(z-\lambda)q(z)$, where $q$ is some polynomial. If $q(\lambda)=0$, then $q$ also can be divided by $z-\lambda$, so $(z-\lambda)^{2}$ divides $p$ and so on.

The largest positive integer $k$ such that $(z-\lambda)^{k}$ divides $p(z)$ is called the multiplicity of the root $\lambda$.

If $\lambda$ is an eigenvalue of an operator (matrix) $A$, then it is a root of the characteristic polynomial $p(z)=\det(A-zI)$. The multiplicity of this root is called the (algebraic) multiplicity of the eigenvalue $\lambda$.

Any polynomial $p(z)=\sum_{k=0}^{n}a_{k}z^{k}$ of degree $n$ has exactly $n$ complex roots, counting multiplicity. The words counting multiplicities mean that if a root has multiplicity $d$ we have to list (count) it $d$ times. In other words, $p$ can be represented as

$$p(z)=a_{n}(z-\lambda_{1})(z-\lambda_{2})\ldots(z-\lambda_{n}).$$

where $\lambda_{1},\lambda_{2},\ldots,\lambda_{n}$ are its complex roots, counting multiplicities.

There is another notion of multiplicity of an eigenvalue: the dimension of the eigenspace $\operatorname{Ker}(A-\lambda I)$ is called geometric multiplicity of the eigenvalue $\lambda$.

Geometric multiplicity is not as widely used as algebraic multiplicity. So, when people say simply “multiplicity” they usually mean algebraic multiplicity.

Let us mention, that algebraic and geometric multiplicities of an eigenvalue can differ.

Computing eigenvalues is equivalent to finding roots of a characteristic polynomial of a matrix (or using some numerical method), which can be quite time consuming. However, there is one particular case, when we can just read eigenvalues off the matrix. Namely

eigenvalues of a triangular matrix (counting multiplicities) are exactly the diagonal entries $a_{1,1},a_{2,2},\ldots,a_{n,n}$

By triangular here we mean either upper or lower triangular matrix. Since a diagonal matrix is a particular case of a triangular matrix (it is both upper and lower triangular

the eigenvalues of a diagonal matrix are its diagonal entries

The proof of the statement about triangular matrices is trivial: we need to subtract $\lambda$ from the diagonal entries of $A$, and use the fact that determinant of a triangular matrix is the product of its diagonal entries. We get the characteristic polynomial

$$\det(A-\lambda I)=(a_{1,1}-\lambda)(a_{2,2}-\lambda)\ldots(a_{n,n}-\lambda)$$

and its roots are exactly $a_{1,1},a_{2,2},\ldots,a_{n,n}$.

Suppose an operator $A$ in a vector space $V$ is such that $V$ has a basis $\mathcal{B}=\mathbf{b}_{1},\mathbf{b}_{2},\ldots,\mathbf{b}_{n}$ of eigenvectors of $A$, with $\lambda_{1},\lambda_{2},\ldots,\lambda_{n}$ being the corresponding eigenvalues. Then the matrix of $A$ in this basis is the diagonal matrix with $\lambda_{1},\lambda_{2},\ldots,\lambda_{n}$ on the diagonal

(4.2.5)

$\displaystyle[A]_{{}_{\scriptstyle\mathcal{B}\mathcal{B}}}=\operatorname{diag}\{\lambda_{1},\lambda_{2},\ldots,\lambda_{n}\}=\left(\begin{array}[]{cccc}\lambda_{1}&&&\\[-10.76385pt] &\lambda_{2}&&{\raisebox{6.45831pt}{\Huge$0$}}\\ &&\ddots&\\[-4.30554pt] {\mbox{\Huge$0$}}&&&\lambda_{n}\end{array}\right).$

On the other hand, if the matrix of an operator $A$ in a basis $\mathcal{B}=\mathbf{b}_{1},\mathbf{b}_{2},\ldots,\mathbf{b}_{n}$ is given by (4.2.5) then trivially $A\mathbf{b}_{k}=\lambda_{k}\mathbf{b}_{k}$, i.e  $\lambda_{k}$ are eigenvalues and $\mathbf{b}_{k}$ are corresponding eigenvectors.

Note that the above reasoning holds for both complex and real vector spaces (and even for vector spaces over arbitrary fields)

Applying the above reasoning to operators in $\mathbb{F}^{n}$ (matrices) we immediately get the following theorem. Note, that while in this book $\mathbb{F}$ is either $\mathbb{C}$ or $\mathbb{R}$, this theorem holds for an arbitrary field $\mathbb{F}$.

As we discussed above, a diagonalizable operator $A:V\to V$ has exactly $n=\dim V$ eigenvalues (counting multiplicities). Any operator in a complex vector space $V$ has $n$ eigenvalues (counting multiplicities); an operator in a real space, on the other hand, could have no real eigenvalues.

We will, as it is customary in the spectral theory, treat real matrices as operators in the complex space $\mathbb{C}^{n}$, thus allowing complex eigenvalues and eigenvectors. Unless otherwise specified we will mean by the diagonalization of a matrix its complex diagonalization, i.e. a representation $A=SDS^{-1}$ where matrices $S$ and $D$ can have complex entries.

The question when a real matrix admits a real diagonalization ($A=SDS^{-1}$ with real matrices $S$ and $D$) is in fact a very simple one, see Theorem 4.2.9 below.

Let the matrix of an operator $A$ in a basis $\mathcal{B}=\mathbf{b}_{1},\mathbf{b}_{2},\ldots,\mathbf{b}_{n}$ is a diagonal one given by (4.2.5). Then it is easy to find an $N$th power of the operator $A$. Namely, the matrix of $A^{N}$ in the basis $\mathcal{B}$ is

$$[A^{N}]_{{}_{\scriptstyle\mathcal{B}\mathcal{B}}}=\operatorname{diag}\{\lambda^{N}_{1},\lambda^{N}_{2},\ldots,\lambda^{N}_{n}\}=\left(\begin{array}[]{cccc}\lambda_{1}^{N}&&&\\[-10.76385pt] &\lambda_{2}^{N}&&{\raisebox{6.45831pt}{\Huge$0$}}\\ &&\ddots&\\[-4.30554pt] {\mbox{\Huge$0$}}&&&\lambda_{n}^{N}\end{array}\right).$$

Moreover, functions of the operator are also very easy to compute: for example the operator (matrix) exponent $e^{tA}$ is defined as $\displaystyle e^{tA}=I+tA+\frac{t^{2}A^{2}}{2!}+\frac{t^{3}A^{3}}{3!}+\ldots=\sum_{k=0}^{\infty}\frac{t^{k}A^{k}}{k!}$, and its matrix in the basis $\mathcal{B}$ is

$$[e^{tA}]_{{}_{\scriptstyle\mathcal{B}\mathcal{B}}}=\operatorname{diag}\{e^{\lambda_{1}t},e^{\lambda_{2}t},\ldots,e^{\lambda_{n}t}\}=\left(\begin{array}[]{cccc}e^{\lambda_{1}t}&&&\\[-10.76385pt] &e^{\lambda_{2}t}&&{\raisebox{6.45831pt}{\Huge$0$}}\\ &&\ddots&\\[-4.30554pt] {\mbox{\Huge$0$}}&&&e^{\lambda_{n}t}\end{array}\right).$$

Let now $A$ be an operator in $\mathbb{F}^{n}$. To find the matrices of the operators $A^{N}$ and $e^{tA}$ in the standard basis $\mathcal{S}$, we need to recall that the change of coordinate matrix $[I]_{{}_{\scriptstyle\mathcal{S}\mathcal{B}}}$ is the matrix with columns $\mathbf{b}_{1},\mathbf{b}_{2},\ldots,\mathbf{b}_{n}$. Let us call this matrix $S$, then according to the change of coordinates formula we have

$$A=[A]_{{}_{\scriptstyle\mathcal{S}\mathcal{S}}}=S\left(\begin{array}[]{cccc}\lambda_{1}&&&\\[-10.76385pt] &\lambda_{2}&&{\raisebox{6.45831pt}{\Huge$0$}}\\ &&\ddots&\\[-4.30554pt] {\mbox{\Huge$0$}}&&&\lambda_{n}\end{array}\right)S^{-1}=SDS^{-1},$$

where we use $D$ for the diagonal matrix in the middle.

Similarly

$$A^{N}=SD^{N}S^{-1}=S\left(\begin{array}[]{cccc}\lambda_{1}^{N}&&&\\[-10.76385pt] &\lambda_{2}^{N}&&{\raisebox{6.45831pt}{\Huge$0$}}\\ &&\ddots&\\[-4.30554pt] {\mbox{\Huge$0$}}&&&\lambda_{n}^{N}\end{array}\right)S^{-1}\ ,$$

and similarly for $e^{tA}$.

Another way of thinking about powers (or other functions) of diagonalizable operators is to see that if operator $A$ can be represented as $A=SDS^{-1}$, then

$$A^{N}=\underbrace{(SDS^{-1})(SDS^{-1})\ldots(SDS^{-1})}_{N\text{ times}}=SD^{N}S^{-1}$$

and it is easy to compute the $N$th power of a diagonal matrix.

We now present very simple sufficient condition for an operator to be diagonalizable, see Corollary 4.2.3 below.

While very simple, this is a very important statement, and it will be used a lot! Please remember it.

To describe diagonalizable operators we need to introduce some new definitions.

Let $V_{1},V_{2},\ldots,V_{p}$ be subspaces of a vector space $V$. We say that the system of subspaces is a basis in $V$ if any vector $\mathbf{v}\in V$ admits a unique representation as a sum

(4.2.7)

$$\mathbf{v}=\mathbf{v}_{1}+\mathbf{v}_{2}+\ldots+\mathbf{v}_{p}=\sum_{k=1}^{p}\mathbf{v}_{k},\qquad\mathbf{v}_{k}\in V_{k}.$$

We also say, that a system of subspaces $V_{1},V_{2},\ldots,V_{p}$ is linearly independent if the equation

$$\mathbf{v}_{1}+\mathbf{v}_{2}+\ldots+\mathbf{v}_{p}=\mathbf{0},\qquad\mathbf{v}_{k}\in V_{k}$$

has only trivial solution ($\mathbf{v}_{k}=\mathbf{0}$ $\forall k=1,2,\ldots,p$).

Another way to phrase that is to say that a system of subspaces $V_{1},V_{2},\ldots,V_{p}$ is linearly independent if and only if any system of non-zero vectors $\mathbf{v}_{k}$, where $\mathbf{v}_{k}\in V_{k}$, is linearly independent.

We say that the system of subspaces $V_{1},V_{2},\ldots,V_{p}$ is generating (or complete, or spanning) if any vector $\mathbf{v}\in V$ admits representation as (4.2.7) (not necessarily unique).

There is a simple example of a basis of subspaces. Let $V$ be a vector space with a basis $\mathbf{v}_{1},\mathbf{v}_{2},\ldots,\mathbf{v}_{n}$. Split the set of indices $1,2,\ldots,n$ into $p$ subsets $\Lambda_{1},\Lambda_{2},\ldots,\Lambda_{p}$, and define subspaces $V_{k}:=\operatorname{span}\{\mathbf{v}_{j}:j\in\Lambda_{k}\}$. Clearly the subspaces $V_{k}$ form a basis of $V$.

The following theorem shows that in the finite-dimensional case it is essentially the only possible example of a basis of subspaces.

To prove the theorem we need the following lemma

First of all let us recall a simple necessary condition. Since the eigenvalues (counting multiplicities) of a diagonal matrix $D=\operatorname{diag}\{\lambda_{1},\lambda_{2},\ldots,\lambda_{n}\}$ are exactly $\lambda_{1},\lambda_{2},\ldots,\lambda_{n}$, we see that if an operator $A:V\to V$ is diagonalizable, it has exactly $n=\dim V$ eigenvalues (counting multiplicities).

Theorem below holds for both real and complex vector spaces (and even for spaces over general fields).

The theorem below is, in fact, already proven (it is essentially Theorem 4.2.8 for real spaces). We state it here to summarize the situation with real diagonalization of real matrices.