Stochastic Series - Pt 4 - State Classification in Markov Chains

23 Sep, 2026

The main question we want to answer is:

“As a Markov chain keeps running, what kind of behavior does each state have?”

We are going to start by looking at recurrent and transient states, but first we need to understand the concept of return time.

Return time T

Starting in a specific state, the return time is the number of steps it takes for a Markov chain to return to that state.

Example:

If the weather is rainy today, how many days until we have another rainy day?

T_{y} = min {n \geq 1 : X_{n} = y}

Symbol	Meaning
$T_{y}$	The first return time to state $y$ , when starting in $y$
$X$	The whole Markov chain
$y$	The state we are interested in, in our example Rainy
$n$	Step number, in our example we use days
$X_{n}$	The state of the chain at step/day $n$
$min$	The smallest value in the set
$:$	Such that, or “where”

So $T_{y} = min {n \geq 1 : X_{n} = y}$ can be read as the following:

Give me the smallest step number greater than or equal to 1 where the chain is in state $y$ .

We start counting at 1 because being in state $y$ at step 0 does not count as returning to it. If the chain never returns, we set $T_{y} = \infty$ .

So in code:

# One sequence of states generated by the chain:
X = ["Rainy", "Sunny", "Sunny", "Rainy"]
y = "Rainy"

Ty = min(
    (i for i, x in enumerate(X) if x == y and i >= 1),
    default=float("inf"),
)

# Ty = 3

Ty gives us the index of the first occurrence of “Rainy” after step 0, in our case 3.

If no return appears in the list, this code gives us infinity. But a finite list only tells us that we have not returned yet, not that we will never return.

Recurrent states

A state is considered recurrent if, starting there, we return to it with probability 1.

Consider a chain with only these transitions:

A → B
B → A

When we go from A to B, we are guaranteed to EVENTUALLY return to A.

To express this in math, we first define the return probability:

ρ_{y y} = P_{y} (T_{y} < \infty)

Symbol	Meaning
$ρ_{y y}$	The probability of eventually returning to state $y$ , starting in $y$
$y$	The state we are interested in, in our weather example Rainy
$P_{y}$	Probability given that the chain starts in state $y$
$T_{y}$	The first return time to state $y$
$<$	Less than
$\infty$	Infinity

Read as:

The probability that we eventually return to state Rainy, given that we start in state Rainy.

For a recurrent state, this probability is 1:

ρ_{y y} = 1

We can think of this as a guaranteed return, although the precise mathematical term is “almost surely”, meaning with probability 1.

Transient states

A transient state is a state we can leave without the guarantee of ever returning.

In the diagrams below, each arrow represents a possible transition with positive probability, and there are no other transitions.

flowchart LR
    A <--> B
    B --> C
    C <--> D

In the above flow, A and B communicate with each other, but when entering B there is a chance of transitioning into C. Once there, we can only move between C and D.

So A and B are both transient states, we are not guaranteed to return.

Their return probabilities are therefore less than 1:

ρ_{y y} < 1

Absorbing states

A state $y$ is absorbing when:

p (y, y) = 1

Read as:

The probability of transitioning from state $y$ to state $y$ again in one step is 1.

An absorbing state is also a recurrent state because staying there counts as returning at the next step.

A → A → A → A → A

But recurrent does not mean absorbing. Consider the chain with only these transitions:

A → B
B → A

A isn't absorbing because we can leave it, but it is recurrent because we are guaranteed to come back to it.

Reaching other states

We define the probability of starting in state $x$ and eventually reaching state $y$ as:

ρ_{x y} = P_{x} (T_{y} < \infty)

Read as:

The probability that we will eventually reach state $y$ , given that we start in state $x$ .

Previously we used $ρ_{y y}$ . Now we write $ρ_{x y}$ because we are starting in $x$ and looking for $y$ . When $x \neq y$ , $T_{y}$ is a hitting time rather than a return time, but we can use the same formula.

So $ρ_{x y} > 0$ means that there is a probability greater than 0 that we will reach state $y$ eventually when starting in state $x$ .

This does not require a one-step transition:

x → B → C → y

The above path also gives us $ρ_{x y} > 0$ , as long as each transition along it has positive probability.

Irreducible sets of states

Imagine a set of states, say:

some_states = {"A", "B", "C"}

This set is irreducible if every state in it communicates with every other state in it.

Mathematically:

i \leftrightarrow j for all i, j \in some_states

Symbol	Meaning
$i$	A state
$j$	A state, possibly the same one
$\in$	Belongs to
$\leftrightarrow$	Communicates with, meaning each state can reach the other

It reads:

Pick any two states from some_states. They must be able to reach each other, possibly through other states, with positive probability.

Positive probability means that there is a nonzero chance of reaching the other state, even if that chance is tiny.

flowchart TD
    A <--> B
    A <--> C
    B <--> C

They can all reach each other.

If the set contains all the states in the chain, we call the whole Markov chain irreducible.

Closed sets

A set is closed if there is no chance of transitioning from a state inside the set to a state outside it.

Mathematically:

p (i, j) = 0 for all i \in some_states, j \notin some_states

Symbol	Meaning
$i \in some_states$	State $i$ belongs to the set
$j \notin some_states$	State $j$ does not belong to the set
$p (i, j) = 0$	The probability of transitioning from $i$ to $j$ in one step is zero

It reads:

For every state $i$ inside the set and every state $j$ outside it, the probability of transitioning from $i$ to $j$ is zero.

Therefore, once you're inside a closed set, you cannot escape.

flowchart LR
    A <--> B
    B --> C
    subgraph closed ["Closed set"]
        C <--> D
    end

In the above example, A and B communicate with each other because communication requires reachability in both directions.

Once you enter C, there is no way of coming back to B or A. You can only move between C and D.

So the communicating classes, the groups of states that can all reach each other, are:

{A, B}, which is not closed because we can leave it through B → C.
{C, D}, which is closed because we cannot leave it.

Both classes are irreducible, but only {C, D} is closed.