The development of autonomous detection technology is imperative in the field of construction. The bucket fill factor is one of the main indicators for evaluating the productivity of construction vehicles. Bucket detection is a prerequisite for bucket trajectory planning. However, previous studies have been conducted under ideal environments, a specific single environment, and several normal environments without considering the actual harsh environments at construction sites. Therefore, seven extreme environments are set in this paper to fill this gap, and an effective method is proposed. First, a novel framework for image restoration under extreme environments is proposed. It applies to all tasks conducted by vision on construction sites. Second, a combination of segmentation and classification networks is used for the first time in this area. Multitask learning is used to discover a positive correlation between fill factor estimation and bucket detection. Furthermore, probabilistic methods and transfer learning were introduced, and excellent results were achieved (97.40% accuracy in fill factor estimation and 99.76% accuracy in bucket detection for seven extreme environments).