So, as someone who was once married to a small-potatoes landlord (and was nominally a landlord myself (1990s) until I lost my share in the divorce settlement), I think I have some perspectives to offer.
There are definitely positive points on both sides, but I'll clarify right off the bat that I no longer understand why any individual should be allowed to own dwellings in which they don't actually live.
Pro-landlord side
Yes, some smaller landlords do actually provide a valuable service.
We inherited the care of 4 single-family homes built c1950 and located on 100 acres of undeveloped forest. We were always on call for repairs, and sometimes did the work ourselves; I particularly remember being in a dark crawlspace in subzero temperatures investigating burst pipes...
Before we took over, it had been managed/owned by $spouse's grandmother, who used the revenue to supplement her retirement income and to pay the property taxes on the 100 acres -- in which real-estate developers were becoming increasingly interested, resulting in taxes ballooning by orders of magnitude. She was not wealthy.
Not long after we took over, we got wind of two similar-age houses downtown that were being given away to anyone who could move them. My then-spouse did the legwork to write up and submit a plan, clear space in the woods (in a younger-growth area of the woods), lay in the foundations, book a house-moving service, etc. all within pretty tight regulatory deadlines which clearly favored people with more resources than we had and which did not seem to place any value on historical preservation.
The initial moving funds came from $spouse's mother -- who cut us off when expenses exceeded the original budget, so I paid for much of the rest of the work (pretty much zeroing out the nest-egg I had accumulated over the previous 2-3 years working as an IT contractor).
So, yeah, we did work for the revenue, both directly (repairs, legal legwork) and indirectly (earning the money to hire others).
This isn't always the case, of course, but sometimes it is. I've certainly met other landlords who were very much hands-on and also decent people.
Anti-landlord side
Pretty much every argument you can put up in favor of this system falls apart when you realize that they're all premised on the idea that money is virtue and that people should have to be employed in order to survive.
(The following quotes are not necessarily from anyone in this discussion; they're just arguments I've heard, repeatedly.)
"I should be allowed to receive the rewards of my investment!" -- lucky you if you have that kind of money to invest. (Lucky us, for having the funds available to move those two additional houses.)
"If we didn't have the rent income, we couldn't afford to live!" -- lucky you, being able to afford to live. Why shouldn't everyone be able to do that, even if they're not lucky enough to own rental houses? If there was UBI and universal free healthcare and free higher education, this would all be moot because we'd all be able to afford to live reasonable lives without having to extract rents from people with less money.
In our particular case, I could make the argument "If we didn't rent out these houses, we couldn't afford to pay the taxes on this land!" -- how did we get to own 100 acres of land in the first place? -- and that is a question I can answer: some not-too-distant lawyer ancestor of $spouse's grandmother had effectively been given a much larger tract spanning all the way from those 100 acres right down to the edge of Athens (GA) -- most of which he later sold off.
So again: privilege and luck, and probably stolen land.
(Am I missing any other pro-landlord arguments?)
I hope I'm not coming across as hostile, here; I'm just seeing a bit of nuance possibly slipping through the cracks -- while still being pretty solidly on the side of "landlords shouldn't be a thing".