Ophiopogon japonicus (O. japonicus) is a traditional medicinal herb and also used as a functional food, where carbohydrates are the key component. We aimed to investigate the structure and activity of O. japonicus polysaccharides (OJP) and oligosaccharides (OJO). OJP was mainly composed of fructose with a molecular weight of 4707 Da, while OJO was mainly composed of fructose and glucose with a molecular weight of 1765 Da. Both OJO and OJP exhibited protective effects against barrier function injury in Caenorhabditis elegans. In colitis mice, OJP and OJO exhibited noticeable relief effects, while OJO significantly promoted the production of short-chain fatty acids compared to OJP. Moreover, OJO had a more pronounced effect in promoting the level of norank_f__Muribaculaceae and norank_f__norank_o__Clostridia_UCG-014. Overall, our findings indicate that molecular weight plays an important role in the anti-inflammatory activities of OJO and OJP and suggest their potential development as therapeutics or functional foods.