Given a set of outlier-contaminated 2D-3D line correspondences between the scene and a captured image, we aim to recover the absolute camera pose. This is a fundamental problem in computer vision and robotics, for which many methods have been developed and shown impressive performance, but they fail to simultaneously guarantee high robustness and efficiency. In this letter, we propose an approach that can process high outlier ratios (e.g., 90%) as the state-of-the-art method while achieving a significant efficiency boost, namely, dozens of times faster. The high robustness and efficiency of our approach benefit from a globally optimal camera orientation estimation module, in which we embed an interval stabbing strategy into a customized Branch-and-Bound (BnB) solving framework. While BnB is widely known to be inefficient, this does not apply to our method thanks to the searching acceleration brought by fast interval stabbing. In addition, we investigate the special case where the camera vertical direction is given as priors; we show that this case can be solved by interval stabbing with both high robustness and real-time efficiency. Experiments on both simulated and real-world data demonstrate the robustness and efficiency of our approach.