Did your baby have trouble adjusting to solids?

My son's pediatrician gave the ok to start solids at 4 and a half months. He was doing alright for a while, but these past few weeks when I tried to increase his solids to twice a day, he began vomiting as if he had the stomach flu. There is a possibility that he had a virus, because I was sick for a weekend, however, he vomited multiple times in a half hour or so and then seemed fine after that... a week later, when I tried giving him two solids in a day, it happened again. I went back to one daily, and he was fine. One week later, I tried giving him solids 2x again, and it happened again. We've been to the doctor/ER each time and they haven't been much help. He never has a fever or any other symptoms.. just vomiting and occasional fussiness.