If there's no merit in having neutral refs, it does pose the question why many other sports (such as football) go to the added expense of appointing neutral officials for all their internationals rather than just hiring the one who lives nearest the ground. It's a matter of optics. To most people it's 'seen' to be fairer to have a neutral ref. If you appoint match officials that are from one of the countries playing, there are inevitably going to be some people who will accuse the officials of bias at every controversial decision.