This paper investigates the image-based visual servoing (IBVS) manipulator system with unknown hysteresis nonlinearity by utilizing the uncalibrated eye-to-hand configuration. Compared with the conventional IBVS system, the unknown actuator hysteresis nonlinearity, which widely exists in the real physical mechanical systems and significantly limits the performances of the robotic system, is fully considered. The problem of unknown actuator nonlinearity prevents direct input of existing controllers and will result in some challenging difficulties in control design. In this paper, an adaptive IBVS control scheme, which takes the unknown actuator hysteresis into account, is presented. Without the prior knowledge of both the intervals of hysteresis parameters and the bound of the dynamic disturbance term, a novel adaptive algorithm is developed to estimate the unknown parameters on-line. The image tracking errors are guaranteed to converge to a small neighborhood of the origin. Simulations and comparative experiments are carried out to test the performance of the proposed scheme.