Radar is among the most popular sensors in modern Intelligent Transportation Systems (ITSs), enabling weather-robust perception. The orientation and position of the traffic radar relative to the ITS coordinate system are necessary for the perception fusion in ITSs. However, due to the unknown target association, sparseness and noisiness of traffic radar measurements, the robust and accurate extrinsic calibration of traffic radar is challenging. In this paper, we propose a targetless traffic radar calibration method based on GPS to overcome the inconvenience during ITS operation, because the installation of a dedicated calibration target on the highway is impractical and dangerous. On the other hand, the high-precision GPS device installed on the moving vehicle can provide traffic radar with accurate positioning information of the detection target. Furthermore, during the optimization process of extrinsic calibration, we propose a globally optimal registration method, which is robust to noise and outliers in radar measurements, and is called Gaussian Mixture Robust Branch and Bound (GMRBnB). Specifically, we first construct the robust objective function by utilizing the Gaussian Mixture Model (GMM). Then, we derive novel relaxation bounds and present the GMRBnB algorithm that overcomes the susceptibility to local minima and the dependence on initialization of traditional optimization methods. Compared with existing methods, extensive experiments in synthetic and real-world data demonstrate that our method is not only globally optimal, but also more accurate and robust.