You are right in that people seem to be misusing the term “social contract”. The actual definition is not vague, but broad. The point is that a person who wants to live in a society accepts its fundamental tenets as they exist in exchange for society at large letting it live amongst itself.
So one example of a social contract might be the one that the USSR and aligned countries’ societies had, which had the fundamental law of “things will continue to improve for you, but in exchange you must support our politics”. When things stopped improving, people stopped staying out of politics and the society collapsed and reformed.
As far as I understand, the US “social contract” is at least on the level of ideology that if you “pull yourself up by the bootstraps”, you get to “live the American Dream”. When they talk about people “rejecting the social contract”, what they mean is that they don’t think that they are getting what they want out of society at large. If it’s few people that do this, you get criminals and sovcits.
This theory is just to explain why people at large behave differently if they perceive their society as good, just, liveable, honourable and all manner of positive things.
Consider the sentiment of “If you saw someone steal from Walmart, you didn’t, but don’t steal from mom-and-pop shops, they need the money”.
People say this is because since Walmart is perceived as not adhering to the social contract by stealing wages, killing off small businesses and mooching off public money. That means that, unlike mom-and-pop shops, they are fair game and not dishonourable to steal from. Formal law still applies to them, you will go to jail if they catch you all the same, but society will not shun you. In a way if you break the “unwritten law”, then it does not protect you either.