Quality calculation of the demagnetization field plays an important role in the computational micromagnetics. It is a nontrivial challenge to develop a robust and efficient algorithm to handle the requirements from the practical simulations, since the nonlocality of the demagnetization field evaluation and the irregularity of the computational domain. In [C. J. Garcia-Cervera and A. M. Roma, Adaptive mesh refinement for micromagnetics simulations, IEEE Trans. Magn., 42(6) (2006), PP. 1648–1654], the evaluation of the demagnetization field is split into solving two partial differential equations by finite difference scheme and calculating the integrals on the domain boundary. It is this integral who causes the computational complexity of the algorithm O(N4/3). To partially resolve the efficiency issue and to make the solver more flexible on handling the magnet with complicated geometry, we introduce an h-adaptive finite element method for the demagnetization field calculations. It can be observed from the numerical results that i). with the finite element discretization, the domain with curved defects can be resolved well, and ii). with the adaptive methods, the total amount of the mesh grids can be reduced significantly to reach the given accuracy, which effectively accelerates the simulations.