Image brightness in color representation has caught active research interests, while its influence to texture features is relatively rarely studied. In this paper, we address the issue of illuminance, or brightness, interference to texture descriptors, especially to the Gabor filter based approaches. Firstly, we reveal the fact that the Gabor filter response linearly drifts with the illuminance. Secondly, the Hodge decomposition is introduced, and a linear regression model is presented to interpret the linear drift phenomenon. An interesting connection with the intrinsic image decomposition is established. Finally, a partial differential equation based method is proposed to compensate the drift issue. Consequently, the texture of the compensated image is enhanced. Extensive experiments are conducted to verify the proposed statements. The proposed linear regression model is verified empirically, and the compensation performances are demonstrated. Classification results on various databases indicate that the compensation can effectively improve the capacity of many existing texture description methods. Especially, the results show that the boosting effect on some deep learning based methods is also valid.