First, it’s not 5x; it’s five orders of magnitude, so that’s 100,000x.
But more importantly, cereal crops are actually not that productive in terms of calorie per unit area of land use. Wheat, for example, produces some 6.4 million calories per acre, which is way less than potatoes (17.8).
Also, no plants require huge variations in their energy input per area than other plants, because they all roughly need what the sun provides. Some plants have different growing seasons, and some can tolerate lower light conditions, but we’re talking about factors of like 1 or 2 here, not 100,000.
The reason grains support civilizations is that they scale as an agricultural practice, and they give you a very convenient resulting food. They’re easy to farm because they’re literally grass, plus their grains can be dried, stored, and transported easily, and they happen to make a reasonably nutritious staple for a diet. Their main labor input is the initial seeding, the harvest, and then the processing (threshing, milling, etc.), all of which scale much better than say picking tomatoes, blueberries, or apples by hand out in the field; this was true back in the day, and is even more true now with modern mechanized harvesting. We still pick tomatoes by hand, but our grains are almost grown autonomously at this point.