I generally wait until it would have the most impact.
A couple times handling some mind-controlled guards? Yeah, sure, they’re probably pretty dumb and not going to respond well.
Keep doing it while raiding the BBEGs territory? They’re going to catch on, and it’s going to bite you in the ass. Won’t just throw out a TPK for it, but you’re going to get an in-game warning that they’re catching on, it’s just up to the players to heed it or not.
Though I will spare a party consequences if they use it again in a more clever way than before, like it becomes a step in a greater trap.