In order to determine the frequency response curve and its unstable region of a strongly nonlinear oscillator, a new method is proposed. This method is based on splitting the system parameters and introducing some unknown parameters into the system. The evaluation of the introduced parameters are done by optimizing the cumulative equation error induced by multiple-scales solution. The Duffing oscillator, the Helmholtz-Duffing oscillator and an oscillator with both nonlinear restoring and nonlinear inertial forces are analyzed as examples to reveal the validity of the proposed method. The frequency-response curves and their unstable regions obtained by the conventional multiple-scales method and the proposed method are compared to those obtained by numerical continuation method and harmonic balance method, respectively. The frequency response curves obtained by numerical continuation method are adopted to compared with those obtained by the proposed method and the conventional multiple-scales method. The unstable regions obtained by the harmonic balance method are adopted to examine those obtained by the conventional multiple-scales method and the proposed method. The efficiency of the proposed method is tested by comparing the computational time of each method.