Accurate traffic prediction is one of the most important techniques in building a smart city. Many works, especially deep learning models, have made great progress in traffic prediction based on rich historical data. However, many cities still suffer from the problem of data scarcity in many aspects. Some works use transfer learning to solve this kind of problem, but what and how to transfer is still an important problem. In this article, we propose a novel Cross-city Multi-Granular Adaptive Transfer Learning method named MGAT for traffic prediction with only a few data in the target city. We first use the meta-learning algorithm to train the model on multiple source cities to get a good initialization. And at the same time, the multi-granular regional characteristics of each source city will be obtained based on our model structure. Then we design an Adaptive Transfer module mainly composed of Spatial-Attention and Multi-head Attention mechanism to automatically select the most appropriate features from the multi-granular features trained from multiple source cities, to achieve the best transfer effect. We conduct extensive experiments on two kinds of real-world traffic datasets cross several cities. Experimental results with other state-of-the-art models demonstrate the effectiveness of the proposed model.