How to compute the probability that the MC ever visits state j given that it starts in i for a Discrete Time Markov Chain.For example, for a DTMC with the state space {1,2,3,4}, and the transition...

To compute the probability that the MC ever visits state j given that it starts in i, you can use
the following steps:
1. Identify the state space of the DTMC, which is the set of all states the MC can be in. In
the example you provided, the state space is {1,2,3,4}.
2. Identify the transition probability matrix of the DTMC, which specifies the probabilities
of transitioning from one state to another. In the example you provided, the transition
probability matrix is given as:
$ \begin{bmatrix}
0 & 0.5 & 0.5 & 0 \
0 & 0.4 & 0.3 & 0.3 \
0 & 0.2 & 0.8 & 0 \
0 & 0 & 0 & 1
\end{bmatrix} $
3. Compute the probability that the MC ever visits state j given that it starts in i by using
the following formula:
$ P_{i,j} = \sum_{k=0}^{\infty} P_{i,j}^{(k)} $
where $ P_{i,j}^{(k)} $ is the probability of transitioning from state i to state j in k steps.
For example, to compute the probability that the MC ever visits state 2 given that it starts in
state 1, you would use the following calculation:
$ P_{1,2} = \sum_{k=0}^{\infty} P_{1,2}^{(k)} = \sum_{k=0}^{\infty} (0.5)^k = \frac{1}{1-0.5} = 2 $
This means that the probability that the MC ever visits state 2 given that it starts in state 1 is 2.
Note that in this calculation, we are assuming that the MC is in state 1 at time 0 and we are
considering all possible transitions to state 2 that may occur in the future, regardless of how
many steps it takes to reach state 2.
The first return time $T_i$ is the earliest time point at which the state of the Markov chain
$X_n$ is equal to $i$. This means that $T_i$ is the smallest value of $n$ such that $X_n = i$.
For example, if the state of the Markov chain at time 0 is $i$, then the first return time $T_i$
would be the smallest value of $n$ such that $X_n = i$. This value would represent the first
time...