Shale is a low-permeability, multi-phase, and multi-scale composite material with intrinsic heterogeneity in micro-texture and mineralogical composition. In this paper, relationships between the microscale texture and constituent phases of shale were investigated using SEM (scanning electron microscopy)/BSE (backscatter electron imaging)-EDS (energy-dispersive x-ray spectroscopy) methods, together with grid nanoindentation experiments. The mechanical properties of the constituent phases of the carbonate-rich Longmaxi shale samples were extracted based on the spatial distribution of mineral phases and the indentation interaction volume. We analyze the effects of particle size on the interpreted mechanical properties and establish a characteristic length scale based on a probabilistic analysis. The identified characteristic length for extracting the mechanical properties of constituent mineral phases from the grid nanoindentation technique is about 5.8–11.7 μm, i.e., up to 10 times greater than that proposed in prior research. A multi-scale mechanical model was established with considerations of a self-consistent scheme for granular morphology to link the microscopic characteristics with the multi-scale mechanical properties of shales. The modeling results show that the stiffness and the strength of the homogenized nano-porous illite/quartz aggregates in Longmaxi shale can be assessed from the nano-scale mechanical properties of the mineral phases and the nano-scale porosity. This study paves the way to accessing the mechanical properties of the constituent phases of composite materials based on the properties of their building blocks and provides extensive insights into their complex mechanical behavior, representing a major step towards developing reliable multi-scale models for engineering applications.