Yes, that is exactly where perfect forward secrecy fails in Element. It allows all of the message keys to be downloaded by attacking a single point of failure. Perfect forward secrecy would necessitate that all messages and their encryption keys be completely independent, and each message would need to be broken one-by-one, as each key is completely different. What Element does with their cloud backup solution is it adds a single point of failure that results in every single message being compromised, without physical access to any device. Real perfect forward secrecy would make that impossible, as you have to break the encryption of every message independently (again, ignoring physical access to the device, because the device will always have access to all the messages anyway). It essentially invalidates many of the benefits of using a double-rachet key exchange protocol to begin with, as you can attack a single point of failure that would compromise all messages instead.
Granted, whether or not that matters to you is entirely up to you. I’m just clarifying that Element lacks perfect forward secrecy, so I have an ideological objection to my own personal use of it for anything sensitive, since there are more secure messengers out there (like SimpleX) that do have perfect forward secrecy, and many more security and privacy features (like the whole no user identifiers thing and no server side storage with SimpleX). That does of course come with the tradeoff that you can only use it on one device at a time, but everything is a list of pros and cons. Is anyone going to target you and attack you by attempting to gain access to your cloud backup keys? No, most certainly not. But the fact that it exists as an attack vector to begin with is troubling from a security perspective (again, that’s where SimpleX shines with all data being stored locally, so there is no way to access those messages on demand without physical access to the device). I personally think that the metadata issues are much worse with Matrix from an immediate privacy perspective, as that is an avenue that can be actively exploited in a much easier capacity.
If I understand correctly though, I believe we’re both on the same page. Element is still a much better option than something like Discord, but it is not without its own flaws.