The accurate trajectory prediction of surrounding vehicles is crucial for the sustainability and safety of connected and autonomous vehicles under mixed traffic streams in the real world. The task of trajectory prediction is challenging because there are all kinds of factors affecting the motions of vehicles, such as the individual movements, the ambient driving environment especially road conditions, and the interactions with neighboring vehicles. To resolve the above issues, this work proposes a novel Heterogeneous Context-Aware Graph Convolutional Networks following the Encoder-Decoder architecture, which simultaneously extracts the hidden contexts from individual historical trajectories, varying driving scene, and inter-vehicle interactional behaviors. Specifically, the historical vehicle trajectories are fed into Temporal Convolutional Network to capture the individual context. Besides, a 2-Dimensional Convolutional Network with temporal attention is designed for transforming the scene image stream into compressing scene context. Then a Spatio-Temporal Dynamic Graph Convolutional Networks is devised to model the evolving interactional patterns, which incorporates the acquired individual and scene contexts as the representation of the node. Finally, the aforementioned three contexts are combined and fed into the decoder to produce future trajectories. The proposed model is validated on two real-world datasets which contain various driving scenarios. Results demonstrated that the proposed model outperforms state-of-the-art methods in prediction accuracy and achieves immense stability towards different vehicle states.