Am I crazy or is how I feel valid??

My boyfriend has developed a friendship with a female work colleague whom I have never met & only found out they were "friends" as he tells me, when I received annonymous txt a saying they were actually sleeping together and more than friends. 
When I told him about they txts he denied anything was going on and said it must be down to someone I knew!! 
I have since discovered that they are texting each other regularly and sometimes up to 30 times in one day! 
I have told him my views and concerns and he said they are just friends and the issue is my problem as nothing is going on and the fact that we are together and he comes home every night should be enough to reassure me. 
His phone is his best friend and never leaves his side and also has a passcode. 
I have asked to meet this other woman to ease my fears but have been told it would be embarrassing for him.
Please help
Are my fears irrational??
We have been together since 17 and are now 34 and have been trying for a baby but I am in such a high state of anxiety that I fear after over twelve months of trying to conceive until these issues are resolved I will never relax enough to get pregnant.