confusion with baby's weight and height...

Marie
My 12 month old just had his 12 month old check up. He is 19lbs 4oz, 27 1/2" . He has always been on the smaller side, although he doesn't really look it. His father is 30 years old and has been 120 lbs up until last year, and his height is 5'4, basically short and leanness is a trend in his family.
  At my son's last check up at 9 months old, he was 18.4lbs and 27 1/4". The doctor asked me early on about my husbands height and I'm starting to think he didn't take note of it (why would he 🙄) He's been pushing me to feed him more for quite a while, my son does eat enough, acts normal, very smart. Other than the concern about his growth he's perfectly healthy and fine.
  The doctor ordered me to get a thyroid test, which required blood work, and was a nightmare for my son and I both. They had to stick him twice and ended up not even getting enough blood. They told me to ask the doctor if this was really necessary for him to have this test because they said at this age it is difficult to draw enough blood. 
   I called the doctors office back, of course he is away now, and the nurse said I didn't go to a good enough lab that deals with babies, why they didn't recommend one in the first place is beyond me. I told her the whole predicament and asked her if it was really necessary for all of this, she said to try another test in a few weeks. They want to do it just to be sure. I'm kind of getting fed up with this doctor, for a few more reasons other than this. He's not very thourough, never records things to remember. 
I just want to know more opinions, is my son's growth a concern if he is seemingly healthy and being shorter and thinner runs in his blood line? Should I really put him through the nightmare of blood work again?