Pointers suck in C++. In other languages every single variable is a pointer and it works perfectly with no memory bugs and great performance.
Pass by value often uses too much memory. Especially if you have a bunch of simultaneous functions/threads/etc that all need to access the same value at once. You can get away with it when your memory is a few dozen integers, but when you’re working with gigabytes of media… you need pointers. Some of the code I work with has values so large they don’t even fit in RAM at all, let alone two or three copies of them. Pass by value could mean writing a hundred gigabytes to swap.