Did he not say "gender is a fact" rather than "sex is a fact"? So where you talk about the terms used in the medical community, he isn't making that distinction.
He also said he's "team-TERF." Quite explicitly.
I dunno. Seems like the backlash is justified.
This article seems to sum it up pretty well: