In this paper, a new method is presented for visual tracking of objects in video sequences. The developed method combines self-organizing map neural network, mean shift segmentation and similarity measurement. The self-organizing map quantizes the image samples into a topological space, it compresses information while preserving the most important topological and metric relationships of the primary features. The mean shift will generate segmentation based on the output of the self-organizing map. Then, according to the segmentation results of the new frame and the first frame, a similarity measurement is used to get the most similar image sample to the specified object in the first frame and thus object position in new frame is found. We apply the developed method to track objects in the real-world environment of surveillance videos. Qualitative and quantitative evaluations indicate that the proposed approach present better results than those obtained by a direct method approach.