Magnetization measurements and time-of-flight neutron powder-diffraction studies on the high-temperature (300--980 K) magnetism and crystal structure (321--1200 K) of a pulverized YCrO$_3$ single crystal have been performed. Temperature-dependent inverse magnetic susceptibility coincides with a piecewise linear function with five regimes, with which we fit a Curie-Weiss law and calculate the frustration factor $f$. The fit results indicate a formation of magnetic polarons between 300 and 540 K and a very strong magnetic frustration. By including one factor $\eta$ that represents the degree of spin interactions into the Brillouin function, we can fit well the applied-magnetic-field dependence of magnetization. No structural phase transition was observed from 321 to 1200 K. The average thermal expansions of lattice configurations (\emph{a}, \emph{b}, \emph{c}, and \emph{V}) obey well the Gr$\ddot{\texttt{u}}$neisen approximations with an anomaly appearing around 900 K, implying an isosymmetric structural phase transition, and display an anisotropic character along the crystallographic \emph{a}, \emph{b}, and \emph{c} axes with the incompressibility $K^a_0 > K^c_0 > K^b_0$. It is interesting to find that at 321 K, the local distortion size $\Delta$(O2) $\approx$ 1.96$\Delta$(O1) $\approx$ 4.32$\Delta$(Y) $\approx$ 293.89$\Delta$(Cr). Based on the refined Y-O and Cr-O bond lengths, we deduce the local distortion environments and modes of Y, Cr, O1, and O2 ions. Especially, the Y and O2 ions display obvious atomic displacement and charge subduction, which may shed light on the dielectric property of the YCrO$_3$ compound. Additionally, by comparing Kramers Mn$^{3+}$ with non-Kramers Cr$^{3+}$ ions, it is noted that being a Kramers or non-Kramers ion can strongly affect the local distortion size, whereas, it may not be able to change the detailed distortion mode.