Ground plane context is an essential semantic information in on-road pedestrian detection task. Due to viewpoint geometry constraints, pedestrians only appear in certain regions of the image, which is close to the horizon area of the ground plane. As a result, the lacking to ground plane context information may cause pedestrian detection system suffering from severe false alarm (i.e. high false positive (FP) rate). For Advanced Driver Assistance System (ADAS), high FP rate not only distracts the driver, but also causes frequent unexpected braking to damage the vehicle's hardware. In this paper, a novel pedestrian detection method called ground plane context aggregation network (GPCAnet) is proposed, which integrates ground plane context information into deep learning based detector to drastically reduce the FP rate. The proposed GPCAnet consists of two modules: i) a ground area predication (GAP) branch is appended on top of convolutional feature map of the backbone network, in parallel with existing branches, for region proposal, classification and bounding box regression; ii) based on GAP, a ground-region proposal network (GRPN) is designed to filter FP cases in order to reduce computations. To evaluate the effectiveness of proposed GPCAnet, experiments on day and night on-road pedestrian detection are performed on both visible and far infrared pedestrian detection datasets, e.g. Caltech and SCUT. Experimental results show that GPCAnet achieves better performance than state-of-the-art methods, while drastically reducing FP rate in pedestrian detection.