You keep on conflating two different things.
Did people 'vote to leave' by splitting the Remain or 2nd Referendum vote between Labour, Lib Dems, Greens and the nationalists - who together got the majority of votes in the 2019 election?
It is fair to say that Labour took a hit in 2019 because a significant part of their 'red wall' votes from 2017 and earlier went to the Tories mainly in response to 'Get Brexit done'.
But majority of seats for a Brexit Tory party and a majority of votes for parties who held the opposite position can co-exist - and did!