Markov Chain¶

Definition¶

A discrete-time Markov chain is a stochastic process $(X_n)_{n\ge0}$ on a finite or countable state space $S$ whose next-state probabilities depend on the observed past only through the present state. For every history with positive probability,

\mathbb{P}(X_{n+1}=j\mid X_0=i_0,\ldots,X_n=i) =\mathbb{P}(X_{n+1}=j\mid X_n=i).

The chain is time-homogeneous when these conditional probabilities do not depend on $n$ . Its transition matrix is $\boldsymbol{P}=(p_{ij})$ $(p_{ij})$ , where $p_{ij}=\mathbb{P}(X_{n+1}=j\mid X_n=i)$ $i)$ , $p_{ij}\ge0$ $0$ , and $\displaystyle\sum_{j\in S}p_{ij}=1$ $1$ .

Interpretation¶

The state must retain all information from the past needed to predict the next transition. A rating class alone may be insufficient if next year's premium also depends on how many consecutive claim-free years have elapsed; include that count in the state. The property does not say that successive states are independent.

Key Results¶

For a finite homogeneous chain, let $\boldsymbol{\alpha}_n$ be the row vector of state probabilities. The matrix product sums over possible intermediate states, giving

\boldsymbol{\alpha}_{n+1}=\boldsymbol{\alpha}_n\boldsymbol{P},\qquad \boldsymbol{\alpha}_n=\boldsymbol{\alpha}_0\boldsymbol{P}^n.

The entry $(\boldsymbol{P}^n)_{ij}$ is the probability of reaching $j$ in $n$ steps when starting at $i$ . These powers require homogeneity; a changing transition rule instead uses the ordered product of the successive matrices.

Examples¶

For $S=\{0,1\}$ and

\boldsymbol{P}=\begin{pmatrix}0.8&0.2\\0.3&0.7\end{pmatrix},

starting in state 0 gives next-step probabilities $(0.8,0.2)$ . After two steps the probability of state 1 is $0.8(0.2)+0.2(0.7)=0.30$ $0.30$ .

Sources¶

MIT Introduction to Probability, Lectures 24–25.
Course placement: RUG Stochastic Models, 2026–2027. These foundation notes have no assigned lecture yet.

Prerequisites¶

Aliases: DTMC Symbols: alpha, element of, sum