Visual system provides comprehensive road information for autonomous driving vehicles. Haze adversely affects the quality of driving images captured by onboard cameras, which poses a significant challenge to the safe operation of vehicles relying on pure vision-based autonomous driving solutions. To address the above issues, a lightweight image dehazing algorithm using a multi-scale architecture called LID-Net is proposed. LID-Net consists of Haze Extraction (HE) blocks and Haze Removal (HR) blocks. The HE block captures more haze features by employing a larger receptive field. The HR block introduces different attention weights to different regions to better remove haze of different concentrations. Furthermore, a Brightness Compensation unit is designed to address the issue of reduced brightness in images following dehazing. This unit compensates for the image's brightness without changing the color of the dehazed image. LID-Net is compared with other state-of-the-art methods on two real-world foggy weather datasets. The results indicate that LID-Net outperforms other methods in terms of dehazing effectiveness. LID-Net can efficiently process an image with a resolution of 1280 × 720 at 125 frames per second, which can fully meet the requirements of real-time processing of automatic driving systems.