You're not alone. I get comments like that all of the time. My best friend from high school, whom I've known for 24 years, is always asking me if I'm "done." She only had two kids, and she doesn't understand why I keep having more. People I work with in the Reserves think I keep getting pregnant to keep from deploying, which is not true. I get pregnant because my husband can't! My brothers are worried about me, because I've had some pretty close calls during my births. They want me to stop for my own health. My older sons even ask me if I know how birth control works, which I can assure you is very embarrassing!
Personally, I feel like I am done now. However, my husband is still dragging his heels on the vasectomy, and I can't take any sort of hormonal birth control (I just don't react well to it in my "old" age). I would not be surprised if I get pregnant again in the next few months. I will have to become a hermit if that happens, because I don't want to deal with all of the negativity that I get from people when they learn how many children I have.
I know what you mean about letting go of your childbearing years. I am now old enough to be a grandma, since my oldest is 19 and (I'm assuming) perfectly capable of fathering a child. My second oldest is 17, so I guess he's old enough, too. I have plenty to do in my "old" age, but I still feel sad that I'm probably done having babies of my own.
My advice is just to ignore everyone but your husband. If the two of you want more children, go for it! As long as you aren't having to borrow money from everyone to raise your children, it really isn't their business. Since your youngest isn't even 3 months old yet, you and your husband might want to wait before making any final decisions, especially if either of you are not sure about how you feel regarding more children. (My husband has been saying for years that he's done, but he still hasn't done anything about it. I suspect that he loves being a daddy more than he lets on, or else he's just afraid of the vasectomy.)
Mom to six boys and two girls.
Mom of nine, ages 22 to 2, six boys, three girls.