So he likes to take a break at weekends (unless it is an emergency, I am sure). Everyone has weekend breaks and has for many years.
The issue is, does his age prevent him from attending meetings that are in his calendar? Did it stop him from achieving much more than previous incumbents? George W. Bush spent more than a year of his presidency at Camp David -- 387 days, either entirely or partially.
It looks suspiciously like other politicians are creating the gossip.
https://prologue.blogs.archives.gov/2015/08/11/take-a-break-with-presidential-vacations/