Target tracking has a wide range of applications in intelligent transportation, real-time monitoring, human-computer interaction and other aspects. However, in the tracking process, the target is prone to deformation, occlusion, loss, scale variation, background clutter, illumination variation, etc., which bring great challenges to realize accurate and real-time tracking. Tracking based on Siamese networks promotes the application of deep learning in the field of target tracking, ensuring both accuracy and real-time performance. However, due to its offline training, it is difficult to deal with the fast motion, serious occlusion, loss and deformation of the target during tracking. Therefore, it is very helpful to improve the performance of the Siamese networks by learning new features of the target quickly and updating the target position in time online. The broad learning system (BLS) has a simple network structure, high learning efficiency, and strong feature learning ability. Aiming at the problems of Siamese networks and the characteristics of BLS, a target tracking method based on BLS is proposed. The method combines offline training with fast online learning of new features, which not only adopts the powerful feature representation ability of deep learning, but also skillfully uses the BLS for re-learning and re-detection. The broad re-learning information is used for re-detection when the target tracking appears serious occlusion and so on, so as to change the selection of the Siamese networks search area, solve the problem that the search range cannot meet the fast motion of the target, and improve the adaptability. Experimental results show that the proposed method achieves good results on three challenging datasets and improves the performance of the basic algorithm in difficult scenarios.