Is it bad to wear yoga pants when pregnant?

So, I know that maternity pants exist but they are SO expensive. At least the ones that I have seen are ridiculously spendy! I'm holding off on buying them as long as I can. I've just been wearing my pre pregnancy yoga pants around. They're comfortable for the most part but are getting snug and after I eat, it does feel a little tight. Everyone keeps telling me that I need to wear loose pants but I already feel frumpy enough as it is. I don't really have a bump yet and things just look weird on me right now. I can't really do any damage to the baby right now from wearing my old yoga pants, can I?