some people just have small babies - you could be one of them. the 12 week u/s is reasonably accurate. an earlier u/s would have been more accurate.
my friend was 38 weeks and her baby was 5lb 3 oz. she went into labor on her own. at 18 months, her child is still tiny.
I was still pretty small at 37 weeks with DS and this pregnancy, I am gaining less weight, and seem to be expanding slower, so might be smaller this time. I think if your doctor isn't worried about your size and the progess of the baby, then you shouldn't worry either.