I've accidentally been accused of staring before. I tend to space out and not even focus my eyes, but it can look like I'm staring. That said, yeah, it's not great. I'm a white guy who's frequently in rural parts of Japan and can get stared at sometimes. Moreso if my tattoos are visible, but that's due to their negative stigma still here (I only have a few on my arms).