In image set classification, affine hull based models have achieved good performance. However, these models only focus on holistic images and ignore the local information of images. To incorporate local information, this paper develops a cooperative operator and proposes a cooperative linear regression model (CLRM) for image set classification. Specifically, CLRM first proposes a block partitioning strategy to obtain local information. Image sets and local information are both modeled as affine hulls. Then, a linear regression model embedded with cooperative operator is proposed to incorporate different information and compute the distance between the different affine hulls. Finally, a combined distance measure is proposed to incorporate different information for classification. Extensive experiments on different recognition tasks such as face recognition on image/video based databases and object classification show that the proposed cooperative operator can improve the performance of the model and the proposed CLRM achieves competitive performance compared with the state-of-the-art classifiers.