nipple shield

So in the hospital they gave me a nipple shield to use because baby wasn't latching very well. So now, we're 5 days PP and I would like to get rid of the shield, but I can't get him to latch right on to my nipple. I've been trying each time I feed him, but he just won't. I can put it right in his mouth and he might suck twice, but then starts fussing. And the more I try the more frustrated he gets because he's hungry. Did any of you have this problem? Is there anything else I can do or try? Do I just keep trying or give up and keep using the shield? Help!