If I do a book report based on a book that I picked up from the library, am I violating copyright? If I write a movie review for a newspaper that tells the plot of the film, am I violating copyright?
The first conceptual mistake in this analogy is assuming the LLM entity is "writing". A person or a sentient being writing is still showing signs of intellectual work, which is how the example book report and movie review will not be accused of plagiarism, which is very very basically stealing someone's output but one that is not made legally ownership of (which then brings it to copyright infringement territory).
LLMs are producing text based on statistical probability meaning it is quite literally aping/replicating the aesthetic form of a known genre of textual output, which in these cases are given the legal status of intellectual property. So yes, an LLM-generated textual output that is in the form of a book report or movie review looks the way it does by copying with no creative intent previous works of the genre. It's the same way YouTube video essays get taken down if it's just a collection of movie clips that might sound like a full dialogue. Of course in that example yt clip, if you can argue it's a creative output where an artist is forming a new piece out of a collage of previous media, the rights owner to those movie clips might lose their claim to the said video. You can't make that defence with OpenAI.