Your knowledge might be incorrect, I believed that before too but it seems to have never been used like that, it has always been against the Jews. Word usage is sometimes not aligned with etymology.
Due to the root word Semite, the term is prone to being invoked as a misnomer by those who incorrectly assert that it refers to racist hatred directed at “Semitic people” in spite of the fact that this grouping is an obsolete historical race concept. Likewise, such usage is erroneous; the compound word antisemitismus was first used in print in Germany in 1879[17] as a “scientific-sounding term” for Judenhass (lit. ‘Jew-hatred’),[18][19][20][21][22] and it has since been used to refer to anti-Jewish sentiment alone.[18][23][24]