The correct answer is $\boxed{\frac{3}{2}}$.
The angular acceleration is given by $\alpha = \frac{F}{I}$, where $F$ is the tangential force and $I$ is the moment of inertia. In this case, $F = 5 \text{ N}$ and $I = 20 \text{ kg.m}^2$, so $\alpha = \frac{5 \text{ N}}{20 \text{ kg.m}^2} = \frac{1}{4} \text{ rad/s}^2$.
The angular velocity is given by $\omega = \omega_0 + \alpha t$, where $\omega_0$ is the initial angular velocity and $t$ is the time. In this case, $\omega_0 = 0$, so $\omega = \frac{1}{4} \text{ rad/s}^2 \times 3 \text{ s} = \frac{3}{2} \text{ rad/s}$.
Therefore, the increase in angular velocity in radian per second is $\boxed{\frac{3}{2}}$.
Here is a brief explanation of each option:
- Option A: $\frac{1}{2}$. This is the angular acceleration of the flywheel. However, the question is asking for the increase in angular velocity, not the angular acceleration.
- Option B: $\frac{3}{2}$. This is the correct answer. The angular acceleration is $\frac{1}{4} \text{ rad/s}^2$, and the time is 3 seconds, so the increase in angular velocity is $\frac{1}{4} \text{ rad/s}^2 \times 3 \text{ s} = \frac{3}{2} \text{ rad/s}$.
- Option C: 2. This is the angular velocity of the flywheel after 3 seconds. However, the question is asking for the increase in angular velocity, not the angular velocity itself.
- Option D: 3. This is the angular acceleration of the flywheel multiplied by the time. However, the question is asking for the increase in angular velocity, not the angular acceleration multiplied by the time.