It’s not actually an individual food that is fattening, it’s an overall diet. So you can have fatty foods and non fatty foods in your diet, and still maintain a calorie deficit or even an overall low fat diet. Plus fat is not the enemy it’s made out to be: you need it for absorbing vitamins and feeling full, among other things.
(Sorry it took me so long to respond, I’m still figuring this out)