Showing affection, means I want sex even when I don't. I don't know what to do

My SO leaves before I wake up and I am home mostly all day alone, so when he is home I am happy to see him. If I give him a kiss, we are good. However, if I do anything else, like hug on him and want to cuddle a bit, or hold his junk, he gets the idea that I want to have sex. But in reality, I just want to hold him. Yes, there are days that I want sex, but not everyday. I am almost 7 months pregnant, all I want to do is sleep, every move i make makes my back hurt so bad which isn't helped by my 2 year old, and I can't roll over with out my pelvis feeling like it is going to snap. I just sometimes want to hold him, or be held (especially be held), but if we do that we have to have sex. But that isn't what I always want. Then he gets passed because I "lead him on" then don't do it.

I just don't know what to do.