Asked #Copilot (formerly #BingChat) a familiar riddle but with numbers changed to make it impossible. It generated the same solution but substituting the numbers so that it ends up with the nonsense claim:
"I have an empty opaque bag. I put two apples and one banana in the bag. I either remove the banana or I remove one apple. I then remove all remaining fruits from the bag. Is it possible to tell what is in the bag now?"
Or maybe it was taught by BingChat/Copilot - isn't Microsoft reportedly using GPT-4? Its solution is even more tortuous and later admits to measuring the wrong amount of water. It never realises it could have stopped after step 1.