The confidence of target detection can be used to evaluate the reliability and risk level of the detected targets and can effective help to exclude the false alarms, but very little investigation was involved in the past. In this letter, a confidence-driven infrared target detection method is proposed. We develop three confidence evaluating methods: (1) the median classification confidence of the cascade classifier; (2) the context confidence based on the number and the confidence of the merged detection rectangles around the detected target; and (3) the contrast confidence based on the difference between the detected target distribution and the around background distribution. The three confidences are combined to form the final confidence of the detected targets. We then use the confidence to refine the localization of the targets. The evaluation using real infrared images demonstrates the good performance of the proposed confidence-driven infrared detection algorithm on both undetected error and false alarm.