Morality is that which directs action to acheive and maintain values. With the pursuit if one means of action, a love occurs, while a simultaneous hatred for other possible actions occurs. Under these terms generally everyone can be labeled a bigot.
Generally those that harm do so because they percieve being harmed, given the subjectivity of percieved harm being a form of harm, the morality behind ending bigotry advances it in certain respects and the problem gets worse as "bigot" becomes a category everyone is bigoted against, thus a downward spiral occurs.
Harm is a matter of perspective and as such does not erase harm from existences but makes it a highly relative interpretation. Given the nature of existence, everything is harmful at one point or another thus leaving the term "harm" an opened ended question that can paradoxicallh magnify itself.
But not to diverge, yes those people where harmed.