I touched on this in previous discussion threads. The retainers are property.
When they “join” the family as retainers, they don’t even get to keep their original names. They insisted on renaming Izutsumi to Asebi.
Their existence is so normalized, Shuro doesn’t give them a second thought. The only one among them he considered a person is Maizuru, as she was his tutor growing up. They formed a bond (this is why Maizuru is so “motherly” towards him), but he grew distant again after learning that his dad was fucking her. The consensuality of which is extremely questionable considering “no” is not an option for a slave.
Shuro has had quite a challenge growing into someone half-decent in an environment where completely fucked up things are normal.