I think we have to draw a picture to understand what this topic is about.
OP, please correct me if I am gerting this wrong, but I believe you are talking about a married couple, whom I shall call Anthony and Barbara, and a third person, male, whom I shall call David and who has an affair with Barbara.
Now, about the suggestion of ChayaTH that "he could not sue you for adultery but he could sue you for defamation", the only possibility of defamation I can see is that if Anthony accuses David publicly of adultery with Barbara, David can sue Anthony for defamation.
Actually, if David is married, to a woman whom I shall call Elisa, there is a second possibility. If Elisa accuses Barbara of adultry with David, Barbara can sue Elisa for defamation.