In the stereo matching task, the 3D convolution network can effectively aggregate the cost volume with the strong representation ability to model the spatial and depth dimensions but with the disadvantage of a high computational cost. In this letter, we revisit the 3D convolution network and its common variant, and then propose the Depth Shift Module (DSM) to model the cost volume in the depth dimension which could imitate the 3D convolution function with the computational complexity of the 2D convolution. The proposed DSM is easy to extend to present 3D cost aggregation methods in stereo matching with less inference time, lower computational complexity, and minor precision loss. Moreover, a novel compact but efficient stereo matching framework named HybridNet is proposed. This framework can hybridize the 2D convolution layer with the proposed DSM to effectively aggregate the cost volume. The proposed HybridNet achieves a better trade-off between the performance, computational complexity, and model size (e.g., 30% less than the size of AANet and 25% less than the size of PSMNet) in public open-source datasets (e.g., Scene Flow and KITTI Stereo 2015). The relevant code is available at https://github.com/GANWANSHUI/HybridNet.