This paper considers the robust optimal control problem for a class of nonlinear systems in the presence of unmodeled dynamics. The adaptive optimal controller design is achieved by the online actor-critic learning, which is robust against the unmodeled dynamics. To deal with the unmodeled dynamics, a dynamical signal with the system state as the input signal is designed to capture the input-to-state stability. In addition to the critic network for value function approximation, a novel robustifying term is developed and introduced into the actor network and applied to the closed-loop system in the presence of the dynamical uncertainty due to the unmodeled dynamics. It is shown that both the actor and critic weights learning converge towards the optimum while guaranteeing the boundedness of all the signals in the closed-loop. Simulation examples are conducted to verify the efficacy of the presented scheme in the end.