I've been doing a bit of reading on this topic since I posted it. I might have found what I'm looking for.
Essentially the reason football's pyramid exists is because it was a merger of a number of different leagues. The football league was considered the primary league in the country so when the football association and football alliance (which were two separate organisations) merged, the football alliance became the second division. This was done to try to win support from fans of football alliance clubs, a case of giving those clubs a chance to play in against the primary football league clubs but saying they have to prove they would be competitive enough, hence P&R. The third division was formed when the football league merged with the Southern league.
An aspect of this seems similar in rugby league. I think it was the second season of our competition, the number of teams was increased by 8 and this is when the Lancashire and Yorkshire leagues were formed which is similar to conferences in a way, though there wasn't a playoff system to decide on a national champion. That brings up another question of why not? Anyway, the Northern Rugby football league was formed in 1901 when members of the county competitions resigned from those leagues arguing that they weren't competitive enough and together those clubs formed another league. It was intended that there would be a playoff between the champions of the county leagues to decide who would be promoted into this new league but the Yorkshire clubs boycotted the new league and so it was decided that only the Lancashire league champion would be promoted. After a lot of shouting from Yorkshire clubs the new league agreed to expand their competition to 15 teams and promote a Yorkshire side as well. With teams leaving the Lancashire league to join this new league tabled a counter proposal of forming a second division. 21 teams from across the two counties applied to join the league and all were admitted which essentially killed off the County leagues. The Lancashire league wound up completely and the Yorkshire league carried on as a junior competition.
So what does all that say? As people have said a conference system is used in a large country and when we had a similar system with the county leagues it was not competitive enough so clubs nationwide formed a new competition. Clubs that were excluded from that formed a second division because their regional competitions were weakened even more from the top clubs buggering off. P&R was put in place so year after year the best clubs could play each other. If some teams declined and there was no P&R then you would have a league that again wasn't as competitive as it could be which makes a lot of sense.
It seems that we did sort of try a conference style system but because of the vastly different size of some clubs, with some based in cities with much more money and support than village teams and with the free market transferring of players which obviously benefits the teams with more money regional competitions simply weren't competitive enough. Contrast that with the NFL which has a set number of teams which are placed in massive metropolises with multi-billionaire owners and stricter rules on transfers and what have you, you are manufacturing a league that is more competitive and so there is little need to keep shuffling teams around to ensure the most competitive are always playing each other.
I can't be bothered to read back through all that. Take a breath goosey!