My biggest gripe as a non native speaker is phrasal verbs.
Unless you know exactly what they mean, you are screwed. You can’t decypher them, there’s no link between the meaning of the component parts and the phrasal verb.
As my English teacher used to tell us jokingly: you should never say: “I get on with my brother, but I get off with my sister”.