No, you'd only have to be the admin. Which is the same at the Fediverse - DMs between two servers can in be seen by the admins of the two servers, should they so desire.
That's not really so different from mainstream social media, the difference here is that the admin is some normal person, not Mark Zuckerberg or Elon Musk or something.
It's absolutely important that people understand this - if you intend for anything to be private, use Matrix or Signal or something. Anything online that is not encrypted is just not truly private. Simple as that.
However, this is also true for any other social media people use. The fediverse is actually kind of neat in that the data is spread out across a bunch of servers, rather than at one central server where the same admins has access to everything.