In this article, we focus on efficient semantic segmentation problem from sequential two-dimensional images, in which all pixels are classified into certain classes for scene understanding. Such problem is challenging because it involves constraints of both spatial and temporal consistencies, which have large difficulties in explicitly determining such structural constraints. Traditionally, such a problem is tackled using structured prediction method, such as conditional random field (CRF). However, pure CRF method suffers from very high complexity in computing high-order potentials and slow performance during inference step, which is unsuitable for efficient video segmentation in real scenario. In this article, a novel feedback-based deep fully convolutional neural network (CNN) is proposed to inherently incorporate spatial context through appending output feedback mechanism. The proposed method has the following contributions: 1) spatial context in images are easily captured through iterative feedback refinement, without the expensive postprocess step such as CRF refinement; 2) easily integrated with generic deep CNN structure; and 3) the inference time is greatly reduced for efficient image segmentation. Compared to current state-of-the-art methods, our proposed method was shown to provide up to 14% better accuracy on semantic segmentation task in challenging Camvid and Cityscapes datasets, while taking up to relatively 980% shorter inference time. The proposed method also shows its effectiveness for real-time road detection task of autonomous driving.