There was a dislike from the Twickenham tendency of league tables. They saw the game as a series of one-offs, organised according to convenience, and they far preferred to play their own kind than a bunch of scary dockers/miners/mill-workers who'd enjoy giving the stockbrokers and coppers a bashing.
Cricket had it's "gentlemen and players" snobbery, but at least they agreed to play alongside each other in meaningful competitions.
The Calcutta Cup was donated with the intention that it be Rugby's equivalent of the FA Cup. Twickenham ignored the donors' wishes and it became just another Home/Five/Six Nations trinket, while Association Football got on with forming leagues and cup competitions, thus conquering the world.