Breasts to swollen for baby to latch?

So I am 4 days PP and my milk supply has started to come in. It has made my boobs very tender and swollen and just engorged! My baby tries so hard to latch on but can't get a good latch because of how swollen they are and gets very frustrated and fussy and after a few minutes refuses to try to latch again. I really want to try to continue to breast feed so I've been pumping and giving him my milk that way but I would like to get him to latch again soon, has anyone else had this problem? What should I do about it?