It was a mildly funny tweet with no connection to racism at all. The blowback is the result of a elements in society that have overplayed the race card (and many other offense / victim narratives) for so long that no matter the shape of the Rorschach ink blot, it's always racist (or some other epithet).
He's a man. He undoubtedly annoys his wife. He understands how being forced to spend so much time together would cause problems. It's likely true that he'd be as annoyed as she would be - maybe more - but that wouldn't play as well, so he made the joke at his gender's expense. No one should be able to predict how this would be offensive, but the victim mentality has become so pervasive it's likely to arise from anything or nothing.
You know, now that I think about it, I'm offended that he assumed that person identified as a woman. ...and that she would be in a cis gender relationship.