Translation please

I'm thinking like 'we probably shouldn't talk now' .....? like it's a difficult time....?

Precise translation depends on the context, but it's a phrase used to defuse potential negative reaction. If taken literally it means, "We surely/probably won't criticize one another.." The word ว่า here means to criticize or admonish, and while กัน literally means 'each other', it often actually means 'me' in spoken Thai. In the phrase ว่ากัน it could mean either, depending on the context.

คงไม่ว่ากัน is basically equivalent to phrases like "I trust you won't hold it against me (if I...)" or "...I'm sure you won't hold that against me." By expressing confidence that they won't respond negatively, it's a way of exerting social pressure to keep them from doing exactly that.

Can you share the context you heard this in?

ช่วงนี้ทำงานเหนื่อยมาก เลยไม่ค่อยมีเวลามาคุยด้วย คงไม่ว่ากันนะ

Here is the context that i received this in, i think the idiomatic translation you gave fits well here.

This is not a word for word translation ... but the above sentence in English would be:

"I'm really busy at the moment with work and don't have much free time to chat. I hope you understand and don't hold it against me."