We propose a novel face representation model, called the polynomial contrast binary patterns (PCBP), based on the polynomial filters, for robust face recognition. It is assumed that the discrete array of pixel values comes about by sampling an underlying smooth surface in an image. The proposed method efficiently estimates the underlying local surface information, which is approximately represented as linear projection coefficients of the pixels in a local patch. The decomposition using polynomial filters can capture rich image information at multiple orientations and frequency bands. This guarantees its robustness to illumination and expression variations. The weighting scheme embeds different discriminative powers of each filter response image. We also propose to carry out a subsequent Fisher linear Discriminant (FLD) on each decomposed image for dimension reduction of features. Our extensive experiments on the public FERET and LFW databases demonstrate that the non-weighted Polynomial contrast binary patterns performs better than most of methods and the weighting scheme further improves the recognition rates. WPCBP+FLD(CD) and WPCBP+FLD(HI) can achieve much competitive or even better recognition performance compared with the state-of-the-art face recognition methods.