The Roman historian Livy writing around 0 AD attests to “the rape of the Sabine Women”, wherein the founders of Rome kidnapped and raped women from surrounding villages, which may just be mythological but is likely a factor in this concept being so prevalent.
As a constrast, though, Madagascar was settled by a mix of Bantu and Indonesian (Austronesian) people, and genetic analysis suggests a founding group of at least 30 women from Indonesia, who almost certainly weren’t stolen from across the Indian Ocean.