Applications to Markov chains

Application to Markov Chains

A Markov chain is a stochastic model describing a sequence of possible events where the probability of each state depends only on the current state (the Markov property). Linear algebra provides the framework for analyzing and computing properties of Markov chains.

Transition Matrix

For a Markov chain with n states, the transition probabilities are encoded in an n × n stochastic matrix P, where Pᵢⱼ is the probability of transitioning from state i to state j. Each row sums to 1.

State Evolution

If x₀ is the initial state probability vector, the state after k steps is:

xₖ = Pᵏx₀

Steady-State Distribution

If the Markov chain is regular (some power of P has all positive entries), the system approaches a unique steady-state vector q such that:

Pq = q

This means q is an eigenvector of P corresponding to eigenvalue λ = 1.

Example

Consider a simple weather model with states "Sunny" and "Rainy":

P = [[0.9, 0.1],
     [0.5, 0.5]]

The steady-state vector solves (P − I)q = 0 with q₁ + q₂ = 1, giving q = [5/6, 1/6]ᵀ. In the long run, 83.3% of days are sunny.

Applications

Markov chains are used in:

• PageRank algorithm (Google search)
• Queueing theory and operations research
• Population genetics
• Financial modeling
• Natural language processing