Matrix factorization (MF), a popular unsupervised learning technique for data representation, has been widely applied in data mining and machine learning. According to different application scenarios, one can impose different constraints on the factorization to find the desired basis, which captures high-level semantics for the given data, and learns the compact representation corresponding to the basis. We note that almost all previous work on MF in data mining has ignored to find such a basis, which can carry high-order semantics in the data. In this work, we propose a novel MF framework called Joint Hypergraph Embedding and Sparse Coding, in which the obtained basis captures high-order semantic information in data. Experimental results on data clustering demonstrate that the proposed method consistently outperforms the other state-of-the-art matrix factorization methods.