Object tracking is a challenging task in computer vision based intelligent transportation systems. Recently, Siamese based object tracking methods have attracted significant attention due to their highly efficient performance. These tracking methods usually train a Siamese network to match the initial target patch of the first frame with candidates in a new frame. In these methods, the offline training of the deep neural network and the online instance searching are effectively combined. However, these methods usually do not include template update or object re-identification, which easily results in the drift problem. In this paper, we propose a novel real-time object tracking method to overcome the above problems by effectively combining a multi-stream Siamese network and a region-based convolutional neural network. Specifically, a novel multi-stream Siamese network is proposed to search the target and update the instance template in a new frame. In addition, a faster region-based convolutional neural network detector is used to perform object re-identification in order to improve the tracking performance by making full use of the object category information. These two networks are tightly coupled to ensure that the proposed tracking method has high efficiency and strong discriminative capability. Experimental results on several object tracking benchmarks show that our tracking method can effectively track vehicles and pedestrians in video sequences by exploiting the object category information. The proposed tracking method achieves real-time operations and outperforms several other state-of-the-art methods.