I think the simplest explanation for why asking an AI for information is so bad is that it doesn't have the concept of not knowing something, it only gives answers, it won't say "I don't know", it just answers, it'll just say whatever, because it doesn't actually know anything, it just says words in response to other words
