Yes, Brexit wasn't a party political issue, but Cameron called the referendum when he didn't have to. Had he not, it may well have led to a Labour government led by Jeremy Corbyn, who despite being anti-EU, would never have called a referendum. Farage would never have won, but he might have wiped out the Tories.
Anyway, it's all history now and maybe off topic.