Currently, the common used vision-based tracking system faces two major challenges: (1) the trade-off between speed and accuracy of the tracker; (2) the robustness of the tracking servo control. In this paper, we propose a new framework composed of a transformer attached to the siamese-type feature extraction networks called Siamese Transformer Network (SiamTrans) to balance the speed and accuracy, avoiding complicated hand-designed components and tedious post-processing of most existing siamese-type trackers with pre-defined anchor boxes or anchor-free schemes. SiamTrans forces the final set of predictions via bipartite matching, significantly reducing hyper-parameters associated with the candidate boxes. Moreover, to enhance the robustness of the servo control, the high-level control part is also redesigned by fusing all the bounding box information and with the Tracking Drift Suppression Strategy (TDSS). The TDSS is mainly used to judge the target's loss. If the target is lost, it will feedback the previous information to reinitialize the tracker to track and update the template patch of SiamTrans, making the whole system more robust. Extensive experiments on visual tracking benchmarks, including GOT-10K, UAV123, demonstrate that SiamTrans achieves competitive performance and runs at 50 FPS. Specifically, SiamTrans outperforms the leading anchor-based tracker SiamRPN++ in the GOT-10K benchmark, confirming its effectiveness and efficiency. Furthermore, SiamTrans is deployed on the embedded device in which the algorithm can be run at 30FPS or 54FPS with TensorRT meeting the real-time requirements. In addition, we design the complete tracking system demo that can accurately track the target for multiple categories. The actual experimental results also show that the whole system is efficient and robust. The demo video link is as follows: https://youtu.be/UK37Q-M9ET4.