But scriptkiddies and automated scans are not a security threat. If they were a legitimate threat to your server, you have bigger problems.
All it does is reduce log chatter.
Anyone actually wanting in would port scan, then try and connect to each port, and quickly identify an SSH port