Supervised learning methods have been used to calculate the stereo matching cost in a lot of literature. These methods need to learn parameters from public datasets with ground truth disparity maps. Due to the heavy workload used to label the ground truth disparities, the available training data are limited, making it difficult to apply these supervised learning methods to practical applications. The two-branch convolutional sparse representation (TCSR) model is proposed in the paper. It learns the convolutional filter bank from stereo image pairs in an unsupervised manner, which reduces the redundancy of the convolution kernels. Based on the TCSR model, an unsupervised stereo matching cost (USMC), which does not rely on the truth ground disparity maps, is designed. A feasible iterative algorithm for the TCSR model is also given and its convergence is proven. Experimental results on four popular data sets and one monocular video clip show that the USMC has higher accuracy and good generalization performance.