In this paper, a tracking control approach for surface vessel is developed based on the new control technique named optimized backstepping (OB), which considers optimization as a backstepping design principle. Since surface vessel systems are modeled by second-order dynamic in strict feedback form, backstepping is an ideal technique for finishing the tracking task. In the backstepping control of surface vessel, the virtual and actual controls are designed to be the optimized solutions of corresponding subsystems, therefore the overall control is optimized. In general, optimization control is designed based on the solution of Hamilton-Jacobi-Bellman equation. However, solving the equation is very difficult or even impossible due to the inherent nonlinearity and complexity. In order to overcome the difficulty, the reinforcement learning (RL) strategy of actor-critic architecture is usually considered, of which the critic and actor are utilized for evaluating the control performance and executing the control behavior, respectively. By employing the actor-critic RL algorithm for both virtual and actual controls of the vessel, it is proven that the desired optimizing and tracking performances can be arrived. Simulation results further demonstrate effectiveness of the proposed surface vessel control.