When the British expelled criminals to Australia they couldn’t return home. Was Australia not a colony? A ton of European immigrants to the American colonies intended their journeys to be one-way trips, and were functionally barred from returning by cost. Does that mean there were no colonies in the Americas?
Besides, throughout history you almost never see settlers leaving en masse when colonial administrations end. Sure, some recent arrivees may turn around, and some administrators who moved there mostly to work in the colonial government may leave, but you really never see the main body of settlers leave. You didn’t even have this in South Africa. They simply have to live under a government where they can’t shoot the locals with impunity.