Organoid, a 3D in vitro cell culture, has high similarities with derived tissues or organs in vivo, which makes it widely used in personalized drug screening. Although organoids play an essential role in drug screening, the existing methods are difficult to accurately evaluate the viability of organoids, making the existing methods still have many limitations in robustness and accuracy. Determination of Adenosine triphosphate (ATP) is a mature way to analyze cell viability, which is commonly used in drug screening. However, ATP bioluminescence technique has an inherent flaw. All living cells will be lysed during ATP determination. Therefore, ATP bioluminescence technique is an end-point method, which only assess cell viability in the current state and unable to evaluate the change trend of cell viability before or after medication. In this paper, we propose a deep learning based framework, OrgaNet, for organoids viability evaluation based on organoid images. It is a straightforward and repeatable solution to evaluate organoid viability, promoting the reliability of drug screening. The OrgaNet consists of three parts: a feature extractor, extracts the representation of organoids; a multi-head classifier, improves feature robustness through supervised learning; a scoring function, measures organoids viability through contrastive learning. Specifically, to optimize our proposed OrgaNet, we constructed the first dedicated dataset, which is annotated by seven experienced experts. Experiments demonstrate that the OrgaNet shows great potential in organoid viability evaluation. The OrgaNet provides another solution to evaluate organoids viability and shows a high correlation compared with ATP bioluminescence technique. Availability: https://github.com/541435721/OrgaNet