Urban sewer systems are vital yet often neglected components of modern infrastructure system. Inspecting these systems is expensive due to labour costs and the need for manual examination by professionals. In addition to the challenges posed by traditional methods, developing deep-learning-based automatic defect detection models requires a vast number of bounding box labels, which are challenging to acquire. To address these gaps, our study introduced the application of Weakly Supervised Object Localization (WSOL) for automated defect localization in sewer pipes. WSOL is a technique that allows for the localization of objects within images using only image-level labels, without the need for precise bounding box annotations. We adopted a state-of-the-art WSOL method that mitigates feature directions with class-specific weights misalignment, enabling more accurate and complete localization of defects. By generating heatmaps from Sewer-ML's image-level annotations, bounding box labels are eliminated, rendering our approach scalable and cost-effective. The proposed WSOL-based approach was validated through five distinct classes of defects and one construction feature, demonstrating the promising localization performance. Our method achieved mean MaxBoxAccV2 scores of 64.33% and 56.89% when using ResNet-50 and VGG-16 backbones, respectively, while also attained classification accuracies of 87.00% for ResNet50 backbone and 83.00% for VGG16 backbone. As a pioneering contribution, our work established a new standard for automated sewer system maintenance, offered a benchmark for the application of WSOL methods using solely image-level annotations in defect localization for urban sewer systems, and further expanded the frontier of weakly supervised learning in critical infrastructural applications.