Linear Algebra Background for Factor Analysis

We now summarize the key concepts from Linear Algebra that are necessary to perform principal component analysis and factor analysis. Additional details can be found in Linear Algebra and Advanced Matrix Topics.

Definition 1: Given a square k ×  k matrix A, an eigenvalue of A is a scalar λ such that det (A – λI) = 0, where I is the k ×  k identity matrix. A non-zero  k × 1 column vector X is an eigenvector which corresponds to eigenvalue λ provided AX = λX.

Observation:  Since any scalar multiple of an eigenvector is also an eigenvector, commonly we consider unit eigenvectors (i.e. an eigenvector whose length is 1). If X = [xi] is an eigenvector corresponding to λ, then X/||X|| is a unit eigenvector corresponding to λ, where ||X|| = \sqrt{\sum_{i=1}^k x_i^2}.

Observation: Eigenvalues and eigenvectors of a square matrix can be constructed in Excel using a variety of approaches. In Excel’s Goal Seek and Solver we show how to find eigenvalues using Excel’s Solver capability; we can then find the corresponding eigenvectors using Gaussian Elimination (although there are some limits to this approach). We also show how to calculate eigenvalues and eigenvectors using QR Factorization (see Orthogonal Vectors and Matrices and Spectral Decomposition).

When you need to find eigenvalues and/or eigenvectors you can use either of these techniques within Excel, but because either method is complicated and time consuming, we suggest that you use following supplemental array functions.

Real Statistics Functions: The Real Statistics Resource Pack provides the following supplemental functions, where R1 is a k × k range in Excel.

eVALUES(R1): Produces an 1 × k array containing the eigenvalues of matrix in range R1. These eigenvalues are listed in decreasing absolute value order.

eVECTORS(R1) : Produces a row with the eigenvalues as for eVALUES(R1). Below each eigenvalue is a unit eigenvector corresponding to this eigenvalue. Thus the output of eVECTORS(R1) must be an (k+1) × k array.

Since the calculation of these functions uses iterative techniques, you can optionally specify the number of iteration used by using eVALUES(R1, iter) and eVECTORS(R1, iter). If the iter parameter is not used then it defaults to 100 iterations.

The eigenvectors produced by eVECTORS(R1) are all orthogonal, as described in Definition 8 of Matrix Operations. See Figure 5 of Principal Component Analysis for an example of the output from the eVECTORS function.

Observation: Every square k × k matrix has at most k (real) eigenvalues (see Eigenvalues and Eigenvectors). If A is symmetric then it has k eigenvalues, although these don’t need to be distinct (see Symmetric Matrices). It turns out that the eigenvalues for covariance and correlation matrices are always non-negative (see Positive Definite Matrices).

Theorem 1 (Spectral Decomposition Theorem): Let A be a symmetric n × n matrix, then A has a spectral decomposition A = CDCT where C is an n × n matrix whose columns are unit eigenvectors C1, …, Cn corresponding to the eigenvalues λ1, …, λn of A and D is the n × n diagonal matrix whose main diagonal consists of λ1, …, λn.

Observation: This is Theorem 1 found in Spectral Decomposition. We will use this theorem to carry out principal component analysis and factor analysis. In fact, the key form of the theorem that we will use is that A can be expressed as

image9167

2 Responses to Linear Algebra Background for Factor Analysis

  1. Richard says:

    Hi Charles,

    I used the eVector function on a 39×39 correlationmatrix. My problem is that it gave me eigenvectors that were neither unitvectors nor orthogonal to each other. As well do I get different eigenvalues when I apply the function eValues. Can you please tell me what I do wrong?

Leave a Reply

Your email address will not be published. Required fields are marked *