It really doesn’t help that the rulebook is filled with bits that could be enforced but aren’t. We are quite lax when it comes to making sure it is accurate and up to date.
The bigger issues, for me, are (a) blaming players who have been fouled for drawing that to the ref’s attention rather than wondering how the ref didn’t punish the foul play or why there is still too much foul play and, (b) people in influential positions who don’t know the rules but pontificate like they did and/or do know the rules but don’t like them and so mock them or downplay the rule breaking.
Overall, the refs have an impossible jobs in many ways. I’m not convinced that many are brilliant but I’ve not seen a truly shocking performance for quite a while.