In recent years, outstanding image-based object detectors were extended for video object detection. Such extension requires to address two challenging problems in videos, namely, scale variation and deformation anomaly. Most image-based works deal with scale variation by resizing the input image for multi-scale training and testing. Such way incurs very high computational cost, which becomes even higher in videos. For deformation anomaly, low-quality RoI feature can be caused by motion blur, video defocus and rare poses, resulting to poor detection accuracy in videos. In this paper, an end-to-end scale-aware coupling-structure network (SCNet) focusing on the two issues is proposed for video object detection with high accuracy and affordable computation. In SCNet, a lightweight scale-aware module is structured to flexibly model the object scale variation, which mainly consists of a set of dilated convolutional layers with parameter constraint. Furthermore, a coupling-structure RoI (region of interest) module is designed to extract robust RoI feature with position-sensitive and context-sensitive information for accuracy improvement. Besides, the feature aggregation strategy is simplified for efficiency. Experiments are conducted on the ImageNet VID dataset. SCNet achieves the state-of-the-art detection performance, exactly 79.5% mAP, with 5.9 points improvement compared to the strong single-frame baseline.