Magnetization measurements and time-of-flight neutron powder-diffraction studies on the high-temperature (300–980 K) magnetism and crystal structure (321–1200 K) of a pulverized YCrO3 single crystal have been performed. Temperature-dependent inverse magnetic susceptibility coincides with a piecewise linear function with five regimes, with which we fit a Curie-Weiss law and calculate the frustration factor f. The fit results indicate a formation of magnetic polarons between 300 and 540 K and a very strong magnetic frustration. By including one factor η that represents the degree of spin interactions into the Brillouin function, we can fit well the applied-magnetic-field dependence of magnetization. No structural phase transition was observed from 321 to 1200 K. The average thermal expansions of lattice configurations (a, b, c, and V) obey well the Gr¨uneisen approximations with an anomaly appearing around 900 K, implying an isosymmetric structural phase transition, and display an anisotropic character along the crystallographic a, b, and c axes with the incompressibility Ka0>Kc0>Kb0. It is interesting to find that at 321 K, the local distortion size Δ(O2) ≈1.96Δ(O1) ≈4.32Δ(Y) ≈293.89Δ(Cr). Based on the refined Y-O and Cr-O bond lengths, we deduce the local distortion environments and modes of Y, Cr, O1, and O2 ions. Especially, the Y and O2 ions display obvious atomic displacement and charge subduction, which may shed light on the dielectric property of the YCrO3 compound. Additionally, by comparing Kramers Mn3+ with non-Kramers Cr3+ ions, it is noted that being a Kramers or non-Kramers ion can strongly affect the local distortion size, whereas, it may not be able to change the detailed distortion mode.