I recommend the book Range, by David Epstein. It makes the point that people can be more successful when they bring different skills and backgrounds to a task or job than if they only narrowly focus on one area their whole career. It’s got lots of examples, discusses what kind of environments reward narrow vs broad range, etc., and overall is a strong endorsement of career hopping.
Life is short. If you’re not happy with what you’re doing, you should make a change.