Jump to content

Errors/Completion Rates & Chance of Success


Recommended Posts

There was some discussion of errors/completion rates and the impact on success over on the Penrith vs Melbourne thread. As a bit of a geek, this gave me the motivation I needed to summarise a fraction of the analytics work I've done this year. In short, over the course of a season, completion rates have no correlation with success, but this may differ by team and doesn't necessarily hold on a match-by-match basis. 

On an annual basis, there is no correlation overall between completion rates and win percentage, assessing 129 overall NRL seasons between 2013 and 2020. This does vary significantly by team though, as shown in the table below, with correlations ranging from 0.6 (Parramatta Eels) to -0.5 (South Sydney):

image.png.72f668082efb171952a6e0b0efbd90bd.png

This may be indicative of differences in playing style. To illustrate this point, offloads also show no overall correlation with win %, however these range from 0.78 (Parramatta Eels) to -0.7 (Sydney Roosters). Interestingly, there is a relationship between the two correlations – they are mildly correlated (0.18) – what this means is that either both completion rate and offloads are both correlated with win %, or neither are, again suggesting this is impacted by playing style. It’s also important to note that as this is based off 8 seasons, the individual team correlations may be impacted by a level of statistical noise.

However, on a per game basis, there is a relationship between errors (not completion rate) and margin/win %. This isn’t a strong correlation, but there is a correlation of -0.23 between errors and margin – as errors increase, margin decreases. This is on all NRL matches from 2016 – 2020 and is shown in the table below.

image.png.80768c71dc37149b7c6eee8d6f1ff316.png

There are a couple of distinct bands here (each change of colour represents a new group), within which a reduction in errors wouldn’t see a notable change in win percentage. Generally, reductions in errors would have to be significant to impact team success. 38% of matches the team has 6 – 9 errors, and 36% of matches the team makes 10 – 12 errors, so for the bulk of matches, errors are likely not a distinguishing factor.

What this analysis doesn’t consider is the errors that the other team made, or differences in playing strength between the two teams. It may be that errors only impact win percentage when there is a particular scale of difference between the errors between the two teams. It may also be that errors matter more when playing a team above you, for example.

It also doesn’t consider the type of error, as the data currently doesn’t record that, at least in terms of what is publicly available. There are arguably 5 key categories of errors: - PTB, catch, pass, taken into touch, contact – these could differ significantly in terms of their impact on team success. For example, it would be expected that PTB errors are negatively correlated with win percentage (as there is no strong obvious tradeoff), whereas catch/pass errors may reap rewards through more expansive attack.

It’s also worth noting that correlation analysis is pretty limited as far as value goes. Multiple regression analysis is much more effective (and also far more geeky) – I’ve produced some of these and at NRL level, errors had no predictive power in terms of score margin. They did at Queensland Cup level but wasn’t particularly powerful – each additional error reduced the margin by -0.55.

Congrats to anyone who has made it all the way through what may be the most geeky post ever made on the RL forum! Hopefully at least one person found this interesting. 

  • Like 7
  • Thanks 4
Link to comment
Share on other sites


Has it been a long lockdown mate.

Problem with stats in a complex sport is no one star will be the overriding factor.

Possession and position can be overridden by penalties and errors, and flukes will still win games.

  • Like 1

Visit my photography site www.padge.smugmug.com

Radio 5 Live: Saturday 14 April 2007

Dave Whelan "In Wigan rugby will always be king"

 

This country's wealth was created by men in overalls, it was destroyed by men in suits.

Link to comment
Share on other sites

17 minutes ago, Padge said:

Has it been a long lockdown mate.

Problem with stats in a complex sport is no one star will be the overriding factor.

Possession and position can be overridden by penalties and errors, and flukes will still win games.

I'm in England so I've not been in lockdown for a while! To be honest analytics is my day job so it's just the combining of two of my interests. One day it would be good to do this for work which is part of the intrigue. 

Agree on no single stat being overriding factor. It doesn't help that all stats as they currently stand are more quantity than quality based. However, they're still one useful tool, as long as they aren't viewed as infallible or hard science, given the assumptions often necessary!

Link to comment
Share on other sites

Cheers for this.

Completely agree that errors should be considered qualitatively much more than quantitatively. (I think this was also brought up on the other thread).

I don't particularly mind seeing my teams handing over possession because they failed to execute an attacking play in good field position. We are, after all, still asking the oppo to start their next set deep in their own half. 

I am less happy if we make an error (at/around PTB, in contact/offloading, get put in touch) within 30m of our line. If this happens in consecutive possessions for us, I will then quietly insist that we complete the next set - the only time I ever utter the phrase - as I think (but with no evidence) that errors in consecutive possessions in poor field position have a multiplicative effect. 

As per the above, I try and get players to think about what any error means in the game's context. Is the error important? Can the error be justified? Should it change the way we approach our next possession? If the answer to the first question is No, stop there.

Most errors aren't important - so move on quickly. Most can be justified - if not, think about it very quickly as an individual and move on. Most don't change our approach to the next possession set - but some do and the onfield leaders need to step in here.

Back to the stats, completion rates are a garbage indicator. Errors could be a useful indicator but should be measured better - probably via category and/or field position, or just assign different errors a different statistical weight.

Link to comment
Share on other sites

The hunt for high set completion rates and low error rates has lead the the game becoming much more stale and boring at the top level, far too many coaches goong for risk averse gameplans based on processes, yardage gains per hit up, gain lines and all kinds of other daft metrics. It's why I prefer watching Hull KR and to a lesser extent Castleford as they are allowed to play heads up when the opportunity is there. Mistakes and errors shouldn't be so hated by coaches, without taking a risk or an error then sod all happens in a game, summed up perfectly last weekend by watching Wigan and Warrington huffing and puffing with almost no attacking allowance to move away from the plan. In Wigans case it was a cross field kick to the left wing, that was in. For warrington god only knows what it was.

Give the players more freedom, more allowance to use their skills, vision, flair. We now prefer robots who can play the process, system, other garbage.

  • Like 1
Link to comment
Share on other sites

10 hours ago, Saint 1 said:

There was some discussion of errors/completion rates and the impact on success over on the Penrith vs Melbourne thread....

 

Thank you for this. Very interesting.

RL, which has a cascade of possibilities from every play, at every moment, is not to be compared with baseball, which is more or less built around a long series of set plays, each one with measurable elements.

What Michael Lewis’s book Moneyball shows, though, is that you can collect  and work from the wrong statistics.

I know you are careful to point out that conclusions from your analysis should be made with great care, but it is really interesting to consider them.

 A high completion rate has become the expected and attempted strategy for success, and maybe it isn’t the best approach.

 Your figures ( and you will have considered them deeply, and your conclusions are worth very much more than mine) perhaps suggest that a low error count could be more significant than completion rate. Pass accurately, catch cleanly, and hold on to the ball in the tackle seems to be a good starting point. Avoid giving away penalties, which I assume is part of the error count. And after that, worry less about completion, and perhaps be more adventurous. 

Pass accurately could be the important part of this. It is really difficult to drop a good pass. A good pass enables you to position the ball well in your arms and torso, gives you more time to position your feet, more opportunity to look up and assess your next action.

 Doing a “Moneyball” on rugby league is deciding where to best spend your recruitement money. So, first choices: an acting-half-back who can pass cleanly and efficiently from the play-the-ball; and a ball distributor who can catch, assess, and pass. But we’ve always known that, haven’t we?

Link to comment
Share on other sites

Thank you for posting this, it's really interesting stuff, to some of us anyway, and also highlights one of the basic scientific principles, that it's just as important to publish evidence that appears to disprove theories as evidence that appears to prove them.

On a more anecdotal note, I recently tracked the error count in two games I watched where I thought it had had an overall impact on the outcome. Both games involved Toulouse in the Championship this year.

The first came when they beat Featherstone 23-6 (disclaimer: I'm a Featherstone fan). The error count was 19 by Featherstone, 9 by Toulouse and as you mention above, the scale of difference in errors may have had a big impact on the outcome (not to mention the fact that 19 errors would be an outlier at this level).

The second game was Oldham v Toulouse. Toulouse made 18 errors but won the game 34-6 (I didn't count Oldham's errors but thought they were no greater than average). However, virtually every formline from this season suggests that Toulouse should be handicapped as 50+ points better than Oldham, and again this outlying error count has to be seen as having an impact on the winning margin (at the risk of stating the obvious).

My probably questionable conclusion was that the impact of the error count amost certainly isn't linear, although other variables, such as the opponents' error count, may neutralise the impact. But it's great to see some properly researched data such as yours.

Link to comment
Share on other sites

It seems to me that the use of statistics in Rugby League coaching has suffered from being shifted from evidence of a performance (good or bad) to a goal in itself.

When completion rates and error counts were first used as evidence of performance I am sure that some coaches and analysts found that in the games that they were successful a low error count or a high completion rate was achieved.  Therefore they executed what they attempted well without incurring errors... hence a win.

But we moved from that to the idea that a high completion rate in itself would therefore lead to a higher chance of success.  But surely only when you successfully execute your attacking plays without error.  If you have a high completion rate without ever threatening the opposition line then you will have no chance of success.  A poor team with no skill or imagination completing at 90% is not going to win next week when they complete at 95% with no skill or imagination.

What I like when I hear coaches like Trent Robinson speak is that they focus on decision making and technique.  In one of the videos that @Saint 1posted on the previous thread, Robinson discussed a fractured defensive line which hindered simultaneous contact from the defenders, or inside defenders turning their hips too much in defense so they weren't able to change direction quickly enough to  respond to a switch play.  Or teams targetting the weak shoulder of the defender with running angles.  These are executable and teachable skills and make players better.  This is what coaches should be focussing on.

If coaches improve players and they execute plays better then the error count will come down while still playing expansive and attacking rugby.  If they simply go straight to the end point which is "let's make fewer errors and we will win the game" then they will be coaching teams with no skill or attacking execution - so high completion rates and defeats.

  • Like 1
Link to comment
Share on other sites

3 hours ago, Archie Gordon said:

I am less happy if we make an error (at/around PTB, in contact/offloading, get put in touch) within 30m of our line. If this happens in consecutive possessions for us, I will then quietly insist that we complete the next set - the only time I ever utter the phrase - as I think (but with no evidence) that errors in consecutive possessions in poor field position have a multiplicative effect. 

I don`t understand why you would be so unhappy with an offload error within your own 30m line, given that a successfully executed offload is one way to relieve the pressure coming out of your own end.

Likewise, if the defence leave a narrow short side wide open a confident player can back himself, beat the markers, head down the touchline, cut back as the cover comes across, hit and spin, find the floor, quick PTB. - Another way to break the shackles from deep.

Of course if either of these go wrong, you might pay a price. But that`s just the risk/reward calculation part of the game.

I recognize the point you make about possible errors on consecutive sets. Although 5 no-frills hit-ups still carries it`s own risk when you`ve been under the pump through a previous error.

Link to comment
Share on other sites

5 minutes ago, unapologetic pedant said:

I don`t understand why you would be so unhappy with an offload error within your own 30m line, given that a successfully executed offload is one way to relieve the pressure coming out of your own end.

Likewise, if the defence leave a narrow short side wide open a confident player can back himself, beat the markers, head down the touchline, cut back as the cover comes across, hit and spin, find the floor, quick PTB. - Another way to break the shackles from deep.

I think this is partly an issue around language. I said I'd be "less happy" with an error made in yardage. But I wouldn't be particularly unhappy with someone offloading and turning over the ball 30m out. Going back to my questions in response to an error: (1) Is it an important error? No - move on from it. As I say, I only worry about errors in our own 30m when they are compounding. 

In yardage, I agree with you that risks can be taken. I'm actually a fan of shifting the ball from DH to a wide first receiver to an edge runner with two long passes if the oppo defensive line is getting up very quickly and has shut us down hard in the middle on tackles 1-3. Errors happen that way - but actually not very often as it's just two passes that we execute a lot in training.

I agree with Dunbar above that a coach's job is not to lay down a bunch of unbreakable rules or set quantitative targets but is instead to help players improve their reading of the game, their decision making, and their tekkers/execution. 

 

  • Like 2
  • Thanks 1
Link to comment
Share on other sites

On 28/09/2021 at 06:45, Archie Gordon said:

Back to the stats, completion rates are a garbage indicator. Errors could be a useful indicator but should be measured better - probably via category and/or field position, or just assign different errors a different statistical weight.

We're on the same page really in terms of how we deal with errors when coaching - I would also share your views on errors impacting exponentially. Minute by minute data would be great for this reason, but alas. 

On 28/09/2021 at 08:15, dkw said:

The hunt for high set completion rates and low error rates has lead the the game becoming much more stale and boring at the top level, far too many coaches goong for risk averse gameplans based on processes, yardage gains per hit up, gain lines and all kinds of other daft metrics. It's why I prefer watching Hull KR and to a lesser extent Castleford as they are allowed to play heads up when the opportunity is there. Mistakes and errors shouldn't be so hated by coaches, without taking a risk or an error then sod all happens in a game, summed up perfectly last weekend by watching Wigan and Warrington huffing and puffing with almost no attacking allowance to move away from the plan. In Wigans case it was a cross field kick to the left wing, that was in. For warrington god only knows what it was.

Give the players more freedom, more allowance to use their skills, vision, flair. We now prefer robots who can play the process, system, other garbage.

People often say this, but out of the 8 semi-finalists across Super League and NRL, how many could really be described as risk averse and boring? Penrith and Saints are probably more conservative than the others, but I still wouldn't say risk averse. 

On 28/09/2021 at 08:56, Cerulean said:

Thank you for this. Very interesting.

RL, which has a cascade of possibilities from every play, at every moment, is not to be compared with baseball, which is more or less built around a long series of set plays, each one with measurable elements.

What Michael Lewis’s book Moneyball shows, though, is that you can collect  and work from the wrong statistics.

I know you are careful to point out that conclusions from your analysis should be made with great care, but it is really interesting to consider them.

 A high completion rate has become the expected and attempted strategy for success, and maybe it isn’t the best approach.

 Your figures ( and you will have considered them deeply, and your conclusions are worth very much more than mine) perhaps suggest that a low error count could be more significant than completion rate. Pass accurately, catch cleanly, and hold on to the ball in the tackle seems to be a good starting point. Avoid giving away penalties, which I assume is part of the error count. And after that, worry less about completion, and perhaps be more adventurous. 

Pass accurately could be the important part of this. It is really difficult to drop a good pass. A good pass enables you to position the ball well in your arms and torso, gives you more time to position your feet, more opportunity to look up and assess your next action.

 Doing a “Moneyball” on rugby league is deciding where to best spend your recruitement money. So, first choices: an acting-half-back who can pass cleanly and efficiently from the play-the-ball; and a ball distributor who can catch, assess, and pass. But we’ve always known that, haven’t we?

In terms of the Moneyball approach, I think first of all the question it attempts to answer is "where do wins come from?", and work backwards from each factor that impacts that. Indeed in Moneyball that was primarily on-base percentage, but in more recent books (Big Data Baseball, The MVP Machine, Astroball) you see this extend to other factors such as fielder positioning, pitching and coaching. I think even placing too much emphasis on errors in recruitment would likely be a waste of time because it's too simplistic a view. With that said, as you highlight, the fundamental skills are the fundamental skills for a reason! 

I've actually built a model which attempts to answer the question I mention above, but obviously there's about 10 relevant statistics - even this is inherently limited because missed tackles is the only defensive stat that enters the model. 

On 28/09/2021 at 09:55, The Phantom Horseman said:

Thank you for posting this, it's really interesting stuff, to some of us anyway, and also highlights one of the basic scientific principles, that it's just as important to publish evidence that appears to disprove theories as evidence that appears to prove them.

On a more anecdotal note, I recently tracked the error count in two games I watched where I thought it had had an overall impact on the outcome. Both games involved Toulouse in the Championship this year.

The first came when they beat Featherstone 23-6 (disclaimer: I'm a Featherstone fan). The error count was 19 by Featherstone, 9 by Toulouse and as you mention above, the scale of difference in errors may have had a big impact on the outcome (not to mention the fact that 19 errors would be an outlier at this level).

The second game was Oldham v Toulouse. Toulouse made 18 errors but won the game 34-6 (I didn't count Oldham's errors but thought they were no greater than average). However, virtually every formline from this season suggests that Toulouse should be handicapped as 50+ points better than Oldham, and again this outlying error count has to be seen as having an impact on the winning margin (at the risk of stating the obvious).

My probably questionable conclusion was that the impact of the error count amost certainly isn't linear, although other variables, such as the opponents' error count, may neutralise the impact. But it's great to see some properly researched data such as yours.

The counterpoint for the Fev Toulouse game is that Fev may have felt they needed to chance their arm in order to win. It's hard to assess errors without understanding the context of the game - perhaps a consideration of passes and offloads may be useful here. Hull KR are a good example of my thinking here - if they play Catalan and pursue an expansive strategy, it may:

- Succeed, passes stick, they make 5 errors and win 24-16 despite being big underdogs

- Fail, passes are dropped, they make 15 errors and lose 36-6

Whereas if they pursue a low risk strategy, they may make 8-12 errors, but lose under all variants of that. 

I think I'm tending towards similar conclusions in terms of non-linearity; I expect the error difference between teams may compound this too. I'm pretty sure I've got the data to assess this at some point so will let you know if I ever do! 

On 28/09/2021 at 10:41, Dunbar said:

It seems to me that the use of statistics in Rugby League coaching has suffered from being shifted from evidence of a performance (good or bad) to a goal in itself.

When completion rates and error counts were first used as evidence of performance I am sure that some coaches and analysts found that in the games that they were successful a low error count or a high completion rate was achieved.  Therefore they executed what they attempted well without incurring errors... hence a win.

But we moved from that to the idea that a high completion rate in itself would therefore lead to a higher chance of success.  But surely only when you successfully execute your attacking plays without error.  If you have a high completion rate without ever threatening the opposition line then you will have no chance of success.  A poor team with no skill or imagination completing at 90% is not going to win next week when they complete at 95% with no skill or imagination.

What I like when I hear coaches like Trent Robinson speak is that they focus on decision making and technique.  In one of the videos that @Saint 1posted on the previous thread, Robinson discussed a fractured defensive line which hindered simultaneous contact from the defenders, or inside defenders turning their hips too much in defense so they weren't able to change direction quickly enough to  respond to a switch play.  Or teams targetting the weak shoulder of the defender with running angles.  These are executable and teachable skills and make players better.  This is what coaches should be focussing on.

If coaches improve players and they execute plays better then the error count will come down while still playing expansive and attacking rugby.  If they simply go straight to the end point which is "let's make fewer errors and we will win the game" then they will be coaching teams with no skill or attacking execution - so high completion rates and defeats.

I think that's a great point on it being a goal in itself. Goodhart's law states "When a measure becomes a target, it ceases to be a good measure." and that is very true in terms of completion rates. The GB tour in 2019 is the strongest example I can think of in that regard. 

On 28/09/2021 at 12:16, Copa said:

Yeah I remember reading about this, and I'd kill to get my hands on their data. While their actual report can't be published, their theory sounds interesting enough. With that said, it feels like there's a risk they're just reporting on two correlated factors, rather than one (lateral movement) predicting the other (tries). This is also what I've done in this thread, but it is a very simplistic approach to stats compared with something like predictive (regression) modelling. For example, do teams move the ball more laterally because they have more ruck speed, or better organisational skills in their spine positions, or more dangerous players on their edges? If so, the lateral movement is relevant but not causative. 

It's a shame the NRL won't publish more data really, a lot of the success in baseball and basketball analytics has been fan-driven and the same could be true in RL. 

  • Like 2
Link to comment
Share on other sites

12 hours ago, Saint 1 said:

People often say this, but out of the 8 semi-finalists across Super League and NRL, how many could really be described as risk averse and boring? Penrith and Saints are probably more conservative than the others, but I still wouldn't say risk averse. 

I would say they definitely are risk averse, compared to several years ago Saints are infinitely more, along with the majority of top league clubs. Its obvious in everything coaches say, its all about set completion, percentage plays, less errors, safer carries, and the huge amount of one up carries 1 pass in we now see in pretty much every SL game.

Link to comment
Share on other sites

16 hours ago, Saint 1 said:

I think that's a great point on it being a goal in itself. Goodhart's law states "When a measure becomes a target, it ceases to be a good measure." and that is very true in terms of completion rates. The GB tour in 2019 is the strongest example I can think of in that regard. 

I have seen something similar play out in my professional career a number of times.  I have instigated and used the Net Promoter score to measures customer experience.  While it has some flaws, overall it is a reasonable measure of service.

When it was introduced it was used by the service professional at point of contact and our scores went up as we tried to deliver a better service.  But it was also gamed by the employees who would put pressure on people to score higher.  so we moved it to being delivered later and anonymised.  Immediately our score went down and there was wailing and gnashing of teeth.

But it entirely missed the point.  The score was the measure of the customer service and it is the service we were looking to improve not the number...the number was just the best metric we had to see if our initiatives were making a difference.

It is the same here.  Teams should be looking to improve their skills and execution.  The error count should go down if they improve these.  But instead, they are just looking at the number.  They are not playing better rugby with a better chance of winning, they are playing worse rugby with a lower error count.  It's crazy if you think about it. 

  • Like 3
Link to comment
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
 Share

  • Recently Browsing   0 members

    • No registered users viewing this page.


×
×
  • Create New...