Visual-based brain-computer interface (BCI) enables people to communicate with others by words and helps professionals recognize targets in large numbers of images. However, the P300 signals evoked by different stimuli such as words or images, may exhibit variability in terms of amplitude and latency, and thus a unified approach for both P300 signals can facilitate BCI application as well as deepen our understanding of the mechanism of P300 generation. In this study, our proposed approach involves using a cascade network structure that combines xDAWN and the classical EEGNet techniques. This network is designed to classify target and non-target stimuli in both P300 speller and rapid serial visual presentation (RSVP) paradigms. The proposed method is capable of recognizing more symbols with fewer repetitions (up to 5 rounds) compared to other models while demonstrating a better information transfer rate (ITR) on dataset II (achieved 17.22 bits/min in the second repetition round) of BCI Competition III. Additionally, our method has the highest unweighted average recall (UAR) performance for both 5 Hz (0.8134±0.0259) and 20 Hz (0.6527±0.0321) RSVP. The results show that the cascade network structure has a better performance between both the P300 Speller and RSVP tasks, manifesting that such a cascade structure is robust enough for dealing with P300-related signals (source code, https://github.com/embneural/Cascade-xDAWN-EEGNet-for-ERP-Detection).