Real-time detection of driver drowsiness is critical to reduce the risk of road accidents and fatalities. Current facial landmark-based methods usually use a two-stage paradigm, where faces and facial landmarks are localized separately. Additionally, most methods can be hindered by challenging conditions, such as night driving or eyes closed. To address these challenges, we present a refined YOLO network named YOLOFaceMark that can simultaneously detect faces and their facial landmarks. Furthermore, we introduce a drowsiness detection model based on facial landmarks. This model utilizes extracted eye and mouth information to identify drowsy states. We optimize the original YOLO components through structural re-parameterization, channel shuffling, and the design of a dual-branch detection head with an implicit module. These enhancements are designed to improve the accuracy while maintaining computational efficiency. We validate the real-time performance and accuracy of YOLOFaceMark on public datasets, including 300W and COFW. Additionally, we conduct further validation to demonstrate our ability to achieve effective and robust drowsiness detection solely based on the facial landmarks detected by YOLOFaceMark.