If I've just added you on LinkedIn and you'd rather just not - that's fine. I don't use it very often and I'm purely doing it on a spread-my-net basis. Reject or ignore away, I will not be offended.
I'm very happy where I am right now so I'm not about to go pressing for opportunities or jobs or anything.
I do also reply nicely to recruiters, because you never know when you will suddenly need options and strings to pull on.