Yeah, I got a Samsung Odyssey HMD cheap (for a VR headset, ~$300) while they were in production and I used my headset to play Half-Life: Alyx, Phasmophobia, and Boneworks. Alyx was very well polished (for a VR game) and nothing came close to that. The rest is varying levels of jank and it will take about 20 mins to set up the headset every time when I could boot up any other game in less than a minute.
And on top of that anything that wasn't those 3 games just didn't feel like it was worth the work to play for me. Boneworks got close to Alyx in terms of polish but if someone was prone to motion sickness at all it would be unplayable. I haven't ever been motion sick in my life but my stomach turned the first time the game "dropped" me down a hole. And Phasmo while it was far more immersive and scary in the headset it just wasn't worth the hassle of setting up the headset only to get a headache in 20 mins of playing because the glare was not great on my headset making dark rooms hard to see in. Better headsets might fix a lot of those issues but the price point, ease of use, and QoL features are not in favor of it ever being more than a party gimmick.
The average person isn't going to want to pay what I did for a few good games that might make them motion sick and a lot of janky garbage. And now most headsets are more expensive than that unless you want the Meta ones that are locked down.