Annoying husbands/boyfriends

I am five weeks and my husband has noticed that my appetite has increased and I sleep more often than usual and longer than usual he makes little comments like I should really start working out before I blow up and it bothers me and annoys me I try to explain this normal at this point for me to be sleepy more often and a little more hungered than usual and that I am not being "lazy"  I wouldn't mind working out but I preferred to wait until around the second trimester because I would feel less paranoid and secure with the stability of the pregnancy . any other ladies dealing with these annoying comments from your significant other ? How do you handle it without resulting to conflict ?