The thing is, the sloppy word choice is still intellectually ableist. I used to use similar language thoughtlessly, and I know first hand that "high intelligence" is a pretty made up and glorified category that doesn't make one immune to bigotry. It is very common to refer to bigots as stupid, just as it is very common to refer to perpetrators of abuse and violence as some kind of crazy, but both are ableist.
I've made a point in getting to know folks of many different walks of life through social justice circles and there are many intellectually disabled people who speak about ableism they face. They face a lot of direct stuff but also it is so common for us non-ID folks to be thoughtlessly, casually ableist with our language.
I've had some positive results from pointing this out to people when they do it. I try to be gentle about it, because I can be not because everyone should be. But I do, and I am. I ask them to please take more care in the future.