I used to be more defensive about this question back when I had just 1 or 2 boys. I really think people are just curious as to whether families with all one gender are going to try for the opposite. That said, I would be offended if it was someone who heavily favored girls and was implying that my boys aren't enough, etc.
Exactly how I feel too. I think most people are just making conversation and don't mean anything by it. Kinda like when you have only one (I had an only for some time) and people ask when you're having another. It depends whether or not I feel the person is judging by making the comment, and of course, alas, some rude people do judge and make hurtful comments, whether total strangers, acquaintances, friends or relatives. If someone was implying that having an only or having all one gender was "deficient" in any way, I would be offended. I mean, it's the same when a married couple gets asked when they're going to have kids - they may not want kids - they may feel happy and complete simply as a couple and that's ok - no one should judge or pity anyone for such life choices or situations.
Btw, good comebacks girls! I will make mental note of them for when I need them (which I hope is not too often once DS2 is born....).