Road segmentation from remote sensing images is a challenging task in capturing weak, long, and irregular road features due to the limited connectivity-preserving modeling capability. In this work, we proposed a U-shaped Fourier-Deformable Convolution Network (FDNet) for road segmentation, which integrates the merits of deformable convolutions and Fourier convolutions compactly. Specifically, a saliency-aware deformable convolution (SD-Conv) layer is proposed for tracing salient road features based on an iterative dynamic offset learning mechanism to grasp extremely tender and weak road objects. Meanwhile, a lightweight global feature extracting module based on spectral convolutions, namely the adaptive Fourier convolution (AF-Conv) layer, is adopted to learn long-range dependency to extract long and continuous road structures. The proposed SD-Conv layer worked in parallel with the AF-Conv layer to construct a basic and compact block to build the U-shaped FDNet model for road segmentation. Furthermore, to maintain the continuity of road objects in complex road conditions, we introduced a topology-oriented loss function based on the Hausdorff distance on the persistence diagram of segmented results, and further combined with softDice loss components for fully supervised training. Our FDNet has been trained and evaluated on two benchmarks, and experimental results show that FDNet achieved state-of-the-art (SOTA) performance. Specifically, it achieved 80.34% on accuracy, 88.42% on precision, and 84.70% on mIoU, respectively, on the Massachusetts dataset, and achieved 99.05% on accuracy, 89.21% on precision, 88.61% on recall, and 81.37% on mIoU, respectively, on the DeepGlobe dataset, outperforming most previous methods on both datasets. Codes are available at: https://github.com/zhoucharming/FDNet.