In this letter, we introduce a new spatio-Temporal feature, namely optical flow energy image (OFEI), for video-based person re-identification. OFEI aims to exploit spatio-Temporally stable regions across frames, which can capture discriminative cues such as human body parts and carry-on stuffs. Furthermore, we propose a novel matching method, denoted by multi-view relevance metric learning with list-wise constraints (mvRMLLC), to integrate the spatio-Temporal (i.e., OFEI) and appearance features. Unlike previous works, mvRMLLC assumes that multiple features are generated from different views with distinct data distributions, while their similarities should be globally consistent. Multiple similarity metrics are then learned and fused by maximizing their global consistency and simultaneously allowing local discrepancies. Extensive experiments on two benchmarks demonstrate that OFEI outperforms the state-of-The-Art spatio-Temporal features, and mvRMLLC could further enhance the overall performance significantly.