Previous research has revealed that the insula, pallidum, thalamus, hippocampus, middle frontal gyrus, and supplementary motor area are activated during odor memory and that the performance of olfactory working memory is affected by the verbalization of odors. However, the neural mechanisms underlying olfactory working memory and the role of verbalization in olfactory working memory are not fully understood. Twenty-nine participants were enrolled in a study to complete olfactory and visual n-back tasks using high- and low-verbalizability stimuli while undergoing fMRI imaging. The behavioral results showed that the participants achieved greater accuracy in the visual rather than olfactory n-back task. We observed increased activation in the precentral gyrus, superior frontal gyrus, middle frontal gyrus, supplementary motor area, and inferior parietal gyrus during olfactory working memory. Interestingly, decreased activation was observed in the olfactory 2-back task versus the 0-back task. Moreover, the left angular gyrus and inferior parietal gyrus were more strongly activated during processing of olfactory working memory using high-verbalizability odors. In conclusion, olfactory working memory engages cross-modal regions to facilitate responses, is involved in the monitoring and manipulation of information during working memory, and boasts a unique activation pattern that is different from that of visual working memory. Semantic information supports the representation of odor information in the working memory system.