work/client issues!

Taylor
I own my own a marketing business and have been planning for a short maternity leave for May (it will be covered by my project manager). One of my (male) clients today let me go cause he said "I want you to get used to being a mom..." followed with lots of other mom/baby struggle comments... I was blind sided by this... has anyone else encountered work issues with pregnancy?! 
If this was a corporate setting I know I could approach this as pregnancy discrimination... but I'm hired as a contractor.