Active fire detection is essential for early warning of wildfires to help suppress and mitigate damage. This study presents an AF-Net model based on object-contextual representations (OCR) for active fire segmentation from very high-resolution (VHR) unmanned aerial vehicles (UAVs) remote sensing images. To efficiently detect heat anomalies in forests from large UAV scenes, we have to handle the class imbalance between small active fire pixels and large-area complex background information. Class imbalance affects the model optimization and makes the training process stuck at a local minimum. Our work aims to address this issue by improving the object-contextual feature representations associated with fire in three ways. First, we employ a grid-based sampling strategy by constraining sampling ranges and reducing background samples. It improves the proportion of foreground pixels from 5.6% to 7.9% and maintains at least one active fire pixel in each sample. Then, we simplify the OCR module to strengthen small object representations related to the active fire using a self-attention unit. The OCR module receives multi-scale pixel representations as input from the HRNet-W48 backbone. Lastly, the weighted binary cross-entropy loss and the Lovász hinge loss are combined to improve the detection accuracy by optimizing the foreground IoU. We evaluate the performance of the proposed AF-Net on one aerial active fire benchmark (FLAME dataset). The proposed framework improves the mIoU score from 78.17% (baseline UNet) to 91.14%.