Yes, you should not be thinking about security in terms of an outside intruder here. Think about untrustworthy or potentially compromised devices.
WiFi smart devices are notorious for calling home, possibly collecting data, even if you’re trying to use them locally.
There have been botnets from unsecured video cameras, and even some compromised from before import.
TVs report back what you’re using them for and when, even playing through hdmi, and some have been caught listening in to your private conversations.
How do you prevent these from happening, or limit what they can do? One way is to put them on a separate vlan without internet access (your HA or other hub can listen on multiple VLANs and be the gatekeeper) and without access to your computers.
That being said, for similar requirements, I found managing the more complex network to be too much hassle, and went back to a simple flat network