The multi-task learning (MTL) methods consider learning a problem together with other related problems simultaneously. The major challenge of MTL is how to selectively screen the shared information. The information of each task must be related to the others, but when sharing information between two unrelated tasks it degenerates the performance of both tasks. To ensure the related problems are related to the main task is the most important point in MTL. In this paper, we will design a novel algorithm to calculate the degrees of relationship among tasks by using a semantical space of features in each task and then build semantical tree to achieve better learning performance. We propose an MTL method under this algorithm which achieves good experimental performance. Our experiments are taken on both image classification and video action recognition, compared with the state-of-the-art MTL methods. Our method proposes good performance in the four public datasets.