With the fast development of online E-commerce Websites and mobile applications, users’ auxiliary information as well as products’ textual information can be easily collected to form a vast amount of training data. Therefore, research efforts are urgently needed to make customized recommendations using such large but sparse data. Deep recommendation model is a natural choice for this research issue. However, most existing approaches try to investigate either user’s auxiliary information such as age and zipcode, or item’s textual information such as product descriptions, reviews or comments. Therefore, it is desired to see whether user’s auxiliary information and item’s textual information could be modeled simultaneously. This paper proposes a novel approach which is essentially a hybrid probabilistic matrix factorization model. Particularly, it has two sub components. One component tries to predict user’s rating scores by capturing user’s personal preferences extracted from auxiliary information. Another component tries to model item’s textual attractiveness to different users via a proposed attention based convolutional neural network. We then propose a global objective function and optimize these two sub components under a unified framework. Extensive experiments are performed on five real-world datasets, i.e., ML-100K, ML-1M, ML-10M, AIV and Amazon sub dataset. The promising experimental results have demonstrated the superiority of our proposed approach when compared with both baseline models and state-of-the-art deep recommendation approaches, i.e., PMF, CDL, CTR, ConvMF, ConvMF+ and D-Attn with respect to RMSE criterion.