In this paper, we propose an environment sensing-aided beam prediction model for smart factory that can be transferred from given environments to a new environment. In particular, we first design a pre-training model that predicts the optimal beam by sensing the present environmental information. When encountering a new environment, it generally requires collecting a large amount of new training data to retrain the model, whose cost severely impedes the application of the designed pre-training model. Therefore, we next propose a transfer learning strategy that fine-tunes the pre-trained model by limited labeled data of the new environment. Simulation results show that when the pre-trained model is fine-tuned by 30 % of labeled data of the new environment, the Top-10 beam prediction accuracy reaches 94%. Moreover, compared with completely re-training the prediction model, the amount of training data and the time cost of the proposed transfer learning strategy reduce 70% and 75% respectively.