Hand tremors may cause some blemishes in precision and stability of a minimally invasive surgery (MIS). To track the tremor signals accurately, there are two main problems left to be settled. First, it is not practical to collect the sample data of tremor in large scale in practical applications. To deal with the hand tremors, a learning method based on small samples sizes and high dimensional input space is needed. Second, the hand tremors have time-varying characteristics. This fact is neglected by traditional learning methods, which could lead to imprecision and instability of a MIS. In this work, a time-sequence-based fuzzy support vector machine adaptive filter (TSF-SVMAF) for tremor cancelling is proposed. The proposed method is based on support vector machine and time series. It is suitable for solving the problem that the inputs are time-varying and the samples are small-scale. To cancel the time-varying hand tremors, different learning-weight-functions are designed for tremor signals with different frequencies. From the simulation results, compared with the existing methods such as back propagation (BP), weighted-frequency Fourier combiner (WFLC) and bandlimited multiple Fourier linear combiner (BMFLC), the proposed method has better performance when learning the time-varying hand tremors with small sample sizes.