Could milk replacer make my goat sick?

I have a 6 week old goat and he was on real goats milk for the first 4 weeks then i slowly introduced milk replacer. He was doing fine. I then moved him outside and he was ok for the first day but they he didn't want to eat anymore and developed diarrhea he was very lethargic and was stretching a lot. I gave him some pepto and he perked up. I gave him his bottle again and again he got sick so i took him off his milk gave him just straight water and the past two days have been adding yogurt to his water, he seems to be doing fine now energy is back and he is his normal playful self. He eats some hay and a bit of grain but i am scared it isn't enough for him. What i would like to know is was it the milk replacer that was making him not feel good and now that he's been off it for a couple days should i try to reintroduce it to him slowly (watered down maybe)