Higher-order singular value decomposition (HOSVD) is one of the most efficient tensor decomposition techniques. It has the salient ability to represent high-dimensional data and extract features. On the other hand, in more recent years, the quaternion has proven to be a very suitable tool for color pixel representation as it can well preserve cross-channel correlation of color channels. Motivated by the advantages of the HOSVD and the quaternion tool, in this paper, we generalize the HOSVD to the quaternion domain and define quaternion-based HOSVD (QHOSVD). Due to the non-commutability of quaternion multiplication, QHOSVD is not a trivial extension of the HOSVD. They have similar but different calculation procedures. Theoretically, QHOSVD is a proper tensor generalization of the quaternion singular value decomposition (QSVD), and a proper quaternion generalization of the standard HOSVD. From the application point of view, the defined QHOSVD can be widely used in various visual data processing with color pixels. As examples, in this paper, we present two applications of the defined QHOSVD in color image processing—multi-focus color image fusion and color image denoising. The experimental results on the two applications respectively demonstrate the competitive performance of the proposed methods over some existing ones.