Also, neural network weights are just a bunch of numbers, and I'm pretty sure data can't be copyrighted.
Just being "a bunch of numbers" doesn't stop it from being a work, it doesn't stop it from being a derivative work, and you absolutely can copyright data -- all digitally encoded works are "just data".
A trained AI is not a measurement of the natural world. It is a thing that has been created from the processing of other things -- in the common sense of it the word, it is derivative of those works. What remains, IMO, is the question of if it would be a work, or something else, and if that something else would be distinct enough from being a work to matter.