I have been in your situation, and felt unfomfortable about it too. And trying to defend
myself and having two sons did not make me feel better , so this IS a tricky
situation to be in. It was at work having coffee with a client who has two
grown up children, one of each gender. And this woman kept saying that
she was so relived that she had at least one daughter cause she never
had any contact with her grown-up son, and you cant ever expect a son
to keep in contact when he is older. That comment did sting - but what she
really said was that she has a not-so good relationship with her son
and isnt that saying something about her more than it says about sons in general?
I think you could have said this to the collegue that made that comment -
"You know, if you get to have two healthy kids of any gender you can consider
yourself very lucky" - maybe sounds just as a cliché but this is true 