As far as I understand if any fediverse instance decided to start having magnet links and stuff like that, eventually it would be taken down by DCMA notices and the like. But what could be done so a community like that could thrive?
No. Magnet links are just that, links.
Some questions:
Would other instances that federate with it get in trouble? They can always feign ignorance.
That depends. Again, magnet links are just links. No one would get in trouble for linking to a magnet link to download #LibreOffice or to download a collection of #eBooks released in the #PublicDomain or #CreativeCommons.
Could an intance which holds magnet links and trackers be backed up every day, so that as soon as one instance is taken down another one takes it's place immediately?
That depends on the people managing the servers. If they want to create backups, then yes, otherwise no.
Could an instance be owned annonymously? Can you register a domain and keep a server running somewhere without repercussions?
Sure. As long as you don't use payment methods that can be traced back to you, and you ensure that your connection is encrypted and you trust your VPN provider. Since you also have to pay for a good VPN provider, that VPN provider should also be accepting a payment method that can not be traced back to you. Not to mention, you should not use your regular computer, regular browser, and definitely avoid mobile phones as you'll always be tracked with it.
Could a lemmy instance live in an onion domain, and still interact with the rest of the fediverse? Would that improve anything regarding the instance being taken down?
Yes. Any #fediverse instance can live in any darknet. Now, if they can connect to those in the Surface Web or Lightnet, all comes down to if the #Darknet they are using offers a bridge back. Or, their server directly connected to the Surface Web instead of passing through a bridge inside their Darknet.
What about running instances in countries that do not enforce piracy laws? Could someone in some random country that does not care maintain an instance, which all of the fediverse can access/comment/contribute?
You can do that. But that instance will more likely get defederated and included in various blocklists or fediblocks.
To sum up. Magnet links are just magnet links. There is nothing illegal about it.
The same with Darknets #TOR and #I2P. There is nothing illegal about it, unless your country passed laws about blocking it.
The real question is, which instances will have issues federating with instances that links to Copyright infringement materials. As linking (not hosting) is vague, the general reaction would more likely be to defederate that instance to be safe.
In the end, that instance's reach will be less and less and less. Probably reach a point wherein it's a lonely island.