No affection allowed!

A little background first I'm living with my boyfriend, and have been for 7 months, we've been together a year. I have a 12yr old that lives with us and he has a 5 & 3 year old that come 3 days a week...........like 4 months ago his ex wife split from her boyfriend of about a year, he was always at her place, showing affection with her in front of the kids (cuddling, kissing) basic innocent stuff. But once they split the kids would ask about him, where he was and why he wasn't coming back, well she didn't want to answer the questions so she avoided it...... Now, like last month, my boyfriend tells me we can no longer show any affection in front of the kids, because God forbid we split and they cant understand....I let it go when he first told me, but again today he felt the need to bring it up an elaborate I guess. I tried to explain to him affection or not, they would probably ask, because I'm around enough..... He says no they won't, because his ex tells him they don't even mention me when I'm not around....... Am I wrong? I think anyone who is missing from the normal routine would cause questions from young children. (not to mention how shitty that all made me feel, when he said they wouldn't notice if I was gone. Like I dont matter, like I don't exist to them, I only make thier meals, wash their clothes, soothe boo-boos, play games, tell stories among other things) I just can't sleep that's why I'm venting, I can't explain my feelings, heart broken, anger towards him, embarrassment, jealously (because she was allowed but now I'm Not cuz she screwed up) , resentment, feeling invisible.... Idk.... Thanks for letting me vent.