Detect highlight event is an important step for semantic-based video retrieval. Hidden conditional random field (HCRF) is a discriminative model, which is effective in fusing observations for event inference. Mid-level semantics and their refinements are more robust than low-level visual features in event detection for learning models. To make full use of the contextual information, two aspects are taken into account during soccer video event detection. The first is parsing video sequences into event clips. The second is fusing the temporal transitions of the mid-level semantics of an event clip to determine the event type. In this study, HCRF is utilised to model the observations of mid-level semantics of an event clip for event detection. Comparisons are made with the dynamic Bayesian networks, hidden Markov model (HMM), enhanced HMM and conditional random field-based event detection approaches. Experimental results show the effectiveness of the proposed method.