Why small minded? I respond to the whole person. I don't see the virtue in compartmentalising.
As I said above I can appreciate the ability and skill of people I don't like but in most cases I won't be buying a ticket to watch them perform. Hitler could have been the greatest water-colourist in history but I wouldn't be a fan even if I acknowledged the skill.
However, in the case of Morrissey it is easy. Don't like the man, don't like the music.