The correct answer is $\boxed{\frac{4}{3}}$.
The expected value of a random variable is the average of its possible values, weighted by their probabilities. In this case, the possible values of $P^2$ are $0, 1, 4, 9$, and their probabilities are $\frac{1}{4}, \frac{2}{4}, \frac{1}{4}, \frac{1}{4}$, respectively. Therefore, the expected value of $P^2$ is
$$\frac{0 \cdot \frac{1}{4} + 1 \cdot \frac{2}{4} + 4 \cdot \frac{1}{4} + 9 \cdot \frac{1}{4}}{4} = \frac{4}{3}.$$
Here is a more detailed explanation of each option:
- Option A: $\frac{2}{3}$. This is the expected value of $P^2$ if the point is randomly selected with uniform probability in the square with corners at $(0, 0), (1, 0), (1, 1), (0, 1)$. The square is a subset of the rectangle, so the expected value of $P^2$ for the rectangle must be at least $\frac{2}{3}$.
- Option B: 1. This is the expected value of $P^2$ if the point is randomly selected with uniform probability in the line segment from $(0, 0)$ to $(1, 2)$. The line segment is a subset of the rectangle, so the expected value of $P^2$ for the rectangle must be at most 1.
- Option C: $\frac{4}{3}$. This is the correct answer.
- Option D: $\frac{5}{3}$. This is the expected value of $P^2$ if the point is randomly selected with uniform probability in the triangle with vertices at $(0, 0), (1, 0), (0, 2)$. The triangle is a subset of the rectangle, so the expected value of $P^2$ for the rectangle must be at most $\frac{5}{3}$.