This isn't a topic I've really engaged in to any large extent myself, however...
Martin - I don't think you can really say whether someone is dead wrong or dead right on the issue. It's a clear matter of opinion which system you prefer as a spectator. I also think the example you cited from the other day was quite and extreme one, with players putting the wrong jersey on. That's pretty rare.
Griff - it does matter to me. I like to know from the off who is playing in which position. It aids my understanding of how the team is functioning when I can see how players in their particular positions are contributing. I find it easier to do this when I can see a positional number on their back.
To give an example, No 17 is the last but one defender out wide, he misses the tackle. He typically plays second row or centre, but can also fill in at loose forward or prop at a stretch. If I'm watching on telly or haven't seen the line up (or indeed remembered where no 17 is playing...) I might have to think, "No 17 is playing 2nd row today, not centre, maybe he shouldn't have had to cover that wide. No 24 should have covered that. He's playing centre today (but was loose forward last week). Incidentally, the no 4, the jersey the centre traditionally wears, is at full back".
I think this is a pretty typical example and I do experience this problem - hence I preferred positional numbering. (fact)