Unfortunately, Haidt is quite wrong about quite a lot. See, for example, here: https://behavioralscientist.org/whats-wrong-with-moral-foundations-theory-and-how-to-get-moral-psychology-right/
In his TED talk about his own research Haidt says that Liberals have two moral foundations and conservatives have five. This is wrong. His own research says, and repeatedly shows, that liberals assign lower priority to three of the five, lower priority for them is not the same as not having them.
Additionally, Haidt, according to his own papers, went into his research with the express intention of finding morals that 'bind communities together' (I didn't put actual quotes there as I can't remember his exact wording), but not all morals are about that. Not surprisingly, the ones that aren't (Care and Fairness) happen to be the two that liberals care about. Just to note, here, that Care and Fairness aren’t about binding communities together, but, if deployed, this the effect they have.
These binding values (In-Group, Authority and Purity) are entrenched in the way that many conservatives engage with their Christianity, Conservatism and Patriotism, but this is also why some Conservatives have more than a little difficulty extending that hand of friendship to non-Christians (and most especially non-believers), non-Conservatives (and most especially progressives) and anyone that doesn't fit their preconceived notion of what an American is (i.e. Christian and Conservative, but often also white). This is why some conservatives have problems with immigrants who are manifestly not all of these things and sometimes are none of these things. The issue, ironically, is the purity of their definitions of these things stops them from being able to extend the hand of friendship.
In point of fact, while many conservatives claim to value Care and Fairness, it is clear that they don't, in part because it's not possible to do so whilst still valuing In-Group, Authority and Purity (hence the inability to stretch the descriptions of American to be more inclusive), and that's over and above the fact that Care and Fairness are orthogonal values that frequently contradict each other.
To illustrate this last point, consider the Christian belief in a God that is infinitely just and infinitely merciful (infinitely fair and infinitely caring, ironically). Justice is to damn someone to hell if they are deserving (the problem of eternal torture for a finite crime is an issue for another discussion). Caring is to recognise that the individual was doing their best with the faculties they had in the context in which they found themselves.
So which wins? Justice: consider the needs of the victims of this individual’s crimes; or, Caring: consider the foibles of a finite and fallen being?
I take your point about moderates being the majority in any given political grouping, however, in America, the moderates are voting for the extremists. The GOP have been increasingly extremist ideologues since Reagan, thanks to the machinations of Falwell and the Moral Majority. That’s very hard to look past for liberals, because bringing religion into politics is antithetical to care and fairness (and contrary to the establishment clause).