In recent years, the optimization of network architecture plays an increasingly important role in the performance improvement of neural networks.We introduce an interactive dual-branch attention mechanism and three different lightweight-oriented strategies to build an accurate and compact residual network model in this work. The channel attention and spatial attention are fused to construct a novel bottleneck to enhance the feature representation ability for accurate performance. Asymmetric convolutions with spatial factorization, channel splitting, depthwise separable convolution with width multiplier adjustment are further combined to compress the parameter size of the attention-driven model for a lightweight and compact residual network named ALResNet. The experimental results of 92.1% top-1 testing accuracy at the inference speed of 14.90 fps on Animals-10 and 89.4% top-1 testing accuracy at the inference speed of 16.21 fps on CIFAR-10, as well as 4.77M parameters and 736.82 MFLOPs, demonstrate that the proposed ALResNet achieves a decent tradeoff between accuracy and computing efficiency for fast inference on resource-limited mobile devices for vision-based tasks.