Crappy grandparents?

What would you do in this situation:

Your kid's grandparents spend no real time with your kid including refusing to even watch them at all. Their excuse: "Not my kid, not my responsibility " and while yes that's true, that's NOT how normal grandparents act. I'm sick of them acting like my son is a burden and I'm pregnant with baby #2 for the reason above, I haven't even told them. I'm tempted to just keep them both away. I don't know what else to do.