In this paper, we propose a robust max-min fairness transmission design for intelligent reflecting surface (IRS)-aided wireless network in the presence of user location uncertainty. In particular, the non-isotropic reflection property for the IRS element is considered. We investigate the joint design of the active transmit beamformer at the base station (BS) and the passive phase shift matrix along with the deployment orientation (facing/pointing direction) of the IRS for maximizing the worst-case minimum signal-to-interference-plus-noise ratio (SINR) received by the users. In order to show the potential gains obtained by adjusting the deployment orientation of the IRS, a single-input-single-output (SISO) system is studied where a closed-form signal-to-noise ratio (SNR) of the user is obtained. For the multi-user case, to solve the resulting non-convex problem, an inexact-alternating-optimization algorithm consisting of a double-loop iteration is proposed. Specifically, in the inner loop, an optimization problem with semi-infinite constraints needs to be solved to increase the worst-case min-SINR compared to the given SINR reference value. We first transform the semi-infinite constraints into linear matrix inequality (LMI) constraints with finite form by applying the Taylor expansion approximation method, the general S-procedure, and the general sign-definiteness lemma. Then an efficient alternating optimization (AO) algorithm based on the two-dimensional search method, negative square penalty (NSP) method, and successive convex approximation (SCA) technique is proposed. While in the outer loop, we update the given SINR reference value as the worst-case minimum SINR obtained after each inner loop iteration. The whole algorithm terminates when the updated SINR reference values converge. Simulation results demonstrate the effectiveness of the proposed algorithm, and also show the additional system performance gain brought by the optimization of the IRS deployment orientation compared to its counterpart with fixed IRS deployment orientation, especially for a smaller IRS element number and a more prominent non-isotropic reflection property of the IRS element.