Legally there are no issues as long as the download option directly downloads from the original host where your application then cuts out the ads. Blocking ads is not illegal. It is neither copyright infringement nor piracy. It might break the ToS but that is ultimately not illegal, and may also be circumventable if carefully implemented.
Invidious does not break the youtube ToS because it doesn’t use the youtube API that the ToS is applied to. Same with Grayjay.