Heat. It's intolerable. Add the modern Rugby suit, and you're gonna have a stroke. Also, the soil is not compatible.
From Wikipedia:
From then on, rugby in India, lingered on at a very low key. Part of the reason for this was that the British preferred to play apart from their colonial subjects, leading to a low take up by the local population. Another reason was the climate, which meant that games would frequently have to be played in the evenings or early morning, which meant that it was not too popular with the colonists themselves.