Companies DO analyze what you say to smart speakers, but only after you have said "ok google, siri, alexa, etc." (or if they mistake something like "ok to go" as "ok google"). I am not aware of a single reputable source claiming smart speakers are always listening.
The reality is that analyzing a constant stream of audio is way less efficient and accurate than simply profiling users based on information such as internet usage, purchase history, political leanings, etc. If you're interested in online privacy device fingerprinting is a fascinating topic to start understanding how companies can determine exactly who you are based solely on information about your device. Then they use web tracking to determine what your interests are, who you associate with, how you spend your time, what your beliefs are, how you can be influenced, etc.
Your smart speaker isn't constantly listening because it doesn't need to. There are far easier ways to build a more accurate profile on you.