The correct answer is $\boxed{\frac{3}{5}}$.
Let $m_1 = 10\text{ kg}$ and $m_2 = 15\text{ kg}$. The initial momentum of the shell is $p_i = m_1v_i = 10\text{ kg}\cdot 100\text{ m/s} = 1000\text{ kg m/s}$. The final momentum of the system is $p_f = m_1v_1 + m_2v_2$. Since the 15 kg mass drops vertically downward, $v_2 = 100\text{ m/s}$ and $m_2v_2 = 1500\text{ kg m/s}$. The 10 kg mass begins to travel at an angle to the horizontal, so $v_1$ has both horizontal and vertical components. The horizontal component of $v_1$ is $v_{1x} = 100\text{ m/s}$, and the vertical component of $v_1$ is $v_{1y}$. The total momentum of the system is conserved, so $p_i = p_f$. This gives us the equation $1000\text{ kg m/s} = 10\text{ kg}v_{1x} + 15\text{ kg}(100\text{ m/s})$. Solving for $v_{1x}$ gives us $v_{1x} = \frac{3}{5}\text{ m/s}$. The angle to the horizontal is $\tan^{-1}(v_{1y}/v_{1x}) = \boxed{\frac{3}{5}}$.
Each option is a possible value for $x$. Option A, $\frac{3}{4}$, is not a possible value because it would give a negative value for $v_{1y}$. Option B, $\frac{4}{5}$, is not a possible value because it would give a value for $v_{1x}$ that is greater than the initial horizontal velocity of the shell. Option C, $\frac{5}{3}$, is not a possible value because it would give a value for $v_{1y}$ that is greater than the initial vertical velocity of the 15 kg mass. Option D, $\frac{3}{5}$, is a possible value because it gives a value for $v_{1y}$ that is less than the initial vertical velocity of the 15 kg mass and a value for $v_{1x}$ that is less than the initial horizontal velocity of the shell.