I don’t think it’s completely ridiculous on its face. Obviously we have some connection to music (as in, we like rhythms, we like making specific sounds with instruments or our voices, we seem to get into the beat etc.), so why shouldn’t it be possible for music known for its complexity to have an effect on us?
It seems it doesn’t, but I don’t think it’s something where you know so “little” about music if you consider it a possibility.