In modern convolutional neural network (CNN)-based object detector, the extracted features are not suitable for multi-scale detection and all the bounding boxes are simply ranked according to their classification scores in nonmaximum suppression (NMS). To address the above problems, we propose a novel one-stage detector named receptive field fusion RetinaNet. First, receptive field fusion module is proposed to extract richer multi-scale features by fusing feature maps of various receptive fields. Second, joint confidence guided NMS is proposed to optimize the post-processing process of object detection, which introduce location confidence in NMS and take joint confidence as the NMS rank basis. According to our experimental results, significant improvement in terms of mean of average precision (mAP) can be achieved on average compared with the state-of-The-Art algorithm.