I have raised this question in the #element-web:matrix.org Matrix room. I got an answer: The use case or feature I have described is still being developed and an experimental version is available as a Beta feature.
On the browser the experimental feature is under: Settings → Labs → New Video rooms
On the desktop app it’s under: Settings → Labs → Join the Beta (joining the Beta enables the “New video rooms”)
From when I tested it, the experimental feature shows the amount of people currently connected to the video/voice call/channel, but not who those users are. I assume that will be added in future.
Either way, thanks for the community for clearing that up for us. As soon as this feature is in the official version, I will try getting my friends from Discord to Matrix. In an initial transition phase probably with Discord-Matrix-bridges.