In the paper we present a new Spatial Adjacent Bag of Features (SABOF) model, in which the spatial information is effectively integrated into the traditional BOF model to enhance the scene and object recognition performance. The SABOF model chooses the frequency of each keyword and the largest frequency of its neighboring pairs to construct the feature histogram. Using the feature histogram whose dimension is only twice larger than that of the original BOF model, the SABOF model drastically enhances the discrimination performance. Combining the Superpixel Adjacent Histogram (SAH) Fulkerson et al., 2009 [12] with multiple segmentations Pantofaru et al., 2008 [33] and Russell et al., 2006 [36], the SABOF method effectively deals with the segmentation and classification of objects with different sizes. Changing the segmentation scale parameter to obtain multiple superpixel segmentations and correspondingly adjusting the neighbor parameters of the SAH method multiple classifiers are trained so that, the SABOF method can fuse multiple results of these classifiers to obtain better classification performance than any single classifier. The superpixel-based conditional random field (CRF) is used to further improve the classification performance. The experimental results of scene classification and of object recognition and localization on classical data sets demonstrate the performance of the proposed model and algorithm. © 2014 Elsevier Inc. All rights reserved.