Only well connected members of the Fascist Regime were benefitting from that (for example, the guy who founded a Coffee Company who got cheap raw materials from the “colonies”, something that at the time was only possible with the authorization of the Fascist Dictator himself).
Most of the rest of the country was incredibly poor and I suspect that the exploitation of the natives in the “colonies” allowed Fascism to keep going a lot longer than otherwise, since Portugal itself (were most people were illiterate subsistence farmers) did not generate much wealth.
It was a Resource Curse kind of situation, with the extra Evil element that the “resource” being exploited to keep the members of the Fascist power circles fat and happy without caring in the least for the welfare of their countrymen, were people in far away lands.