This argument is significantly weaker than most people who voice it are aware, imo.
Our current understanding of the universe, physics, space travel, etc., leads us to believe your statement. The universe is too big, travel takes too long, we’re limited by things like the speed of light, and whatever fuel/energy is required to travel through space.
Speculating here obviously, an intelligent species that is significantly more advanced than us could have a much better understanding of physics and the universe, to an extent that we couldn’t even comprehend. This is the definition of “more advanced”. They could be a little more advanced, or so far beyond us that we could never even comprehend their abilities or motives for anything. Kinda like how you will never be able to teach a chimpanzee calculus, it’s brain simply isn’t capable of learning the concepts, ever, no matter how much time you have to try.
We’ve only really had advanced applied science (according to us, since it’s all we know) for like the last 100 years or so? Splitting the atom, the transistor, etc.
Imagine a species that kept advancing for 100s of thousands or 10s of millions of years? Traveling galaxies could be easy for them, we simply don’t know.
My reasoning above is also similar why I feel like the Fermi peridox is really unsatisfying. We can’t see aliens emitting technosignatures like light or other electromagnetic radiation, or heat signatures or whatever. We’re thinking like humans, expecting to see things exactly as we do them. If an alien species is so far beyond us, they could think communication via photons and electromagnetic radiation is archaic, etc.