The correct answer is D. Only one worker in the factory has exactly five children.
Option A is incorrect because it is possible that some workers in the factory have more than five children, or that some workers have no children.
Option B is incorrect because it is possible that some workers in the factory do not have children.
Option C is incorrect because it is possible that some workers in the factory have exactly five children, while others have more or fewer than five children.
Option E is incorrect because at least one worker in the factory has exactly five children.