Background modeling has played an important role in detecting the foreground for video analysis. In this paper, we presented a novel background modeling method for foreground segmentation. The innovations of the proposed method lie in the joint usage of the pixel-based adaptive segmentation method and the background updating strategy, which is performed in both pixel and object levels. Current pixel-based adaptive segmentation method only updates the background at the pixel level and does not take into account the physical changes of the object, which may result in a series of problems in foreground detection, e.g., a static or low-speed object is updated too fast or merely a partial foreground region is properly detected. To avoid these deficiencies, we used a counter to place the foreground pixels into two categories (illumination and object). The proposed method extracted a correct foreground object by controlling the updating time of the pixels belonging to an object or an illumination region respectively. Extensive experiments showed that our method is more competitive than the state-of-the-art foreground detection methods, particularly in the intermittent object motion scenario. Moreover, we also analyzed the efficiency of our method in different situations to show that the proposed method is available for real-time applications.