I never found any danger there, and I always walked through the poor area late at night to get from bus station to airport. Everyone was very polite as they sat outside their shanties, "Enjoy your holiday sir" was often said as I walked past. (Normally the locals will call you 'sir' or 'boss', no racial slurs at all, they seem to like foreigners)
As for places to live, Borocay would probably be the most expensive.
I used to stay in Barretto (near Olongapo), very nice beaches, lots of foreigners, many 'nightclubs' and resorts. Can't say what it's like now with COVID, might be as locked down as Pattaya. Was no problem to have Aussie meat pies for lunch in the local pubs. Lots of Brits and Australians running most of the resorts.
Free 1 year VISA from the consulate if you are entering with your Filipino wife.
This is Baloy Long Beach just before COVID.