User-generated content is daily produced in social media, as such user interest summarization is critical to distill salient information from massive information. While the interested messages (e.g., tags or posts) from a single user are usually sparse becoming a bottleneck for existing methods, we propose a topic-aware graph-based neural interest summarization method (UGraphNet), enhancing user semantic mining by unearthing potential user relations and jointly learning the latent topic representations of posts that facilitates user interest learning. Experiments on two datasets collected from well-known social media platforms demonstrate the superior performance of our model in the tasks of user interest summarization and item recommendation. Further discussions also show that exploiting the latent topic representations and user relations are conductive to the user automatic language understanding.