IMG Grading System (Many Merged Threads)

[Archie Gordon]

Just now, dkw said:

So how are you quantifying that they have done both things poorly for us, what measureables are you basing it on?

Both exercises have been categorically poor.

The survey was abysmal - dreadful questions, a self-selected sample, opportunity for one person to respond multiple times, etc.

For the grading, I can keep drip feeding examples but everyone knows them by now.

Here's another. Both TO and London scored A grades on fandom. They have very small fan bases indeed. Does anybody really think these two clubs are pretty much in the same space on fandom as Wigan? Indeed, from what I can gather, achieving 1.5-3k fans allowed several clubs to score an A for fandom. 

[gingerjon]

2 minutes ago, Archie Gordon said:

Both exercises have been categorically poor.

The survey was abysmal - dreadful questions, a self-selected sample, opportunity for one person to respond multiple times, etc.

For the grading, I can keep drip feeding examples but everyone knows them by now.

Here's another. Both TO and London scored A grades on fandom. They have very small fan bases indeed. Does anybody really think these two clubs are pretty much in the same space on fandom as Wigan? Indeed, from what I can gather, achieving 1.5-3k fans allowed several clubs to score an A for fandom. 

Assume for this purpose that you are in favour of gradings, and given this is the off season and the only alternative is work, what would your grading system look like?

[Dave T]

2 minutes ago, Archie Gordon said:

Both exercises have been categorically poor.

The survey was abysmal - dreadful questions, a self-selected sample, opportunity for one person to respond multiple times, etc.

For the grading, I can keep drip feeding examples but everyone knows them by now.

Here's another. Both TO and London scored A grades on fandom. They have very small fan bases indeed. Does anybody really think these two clubs are pretty much in the same space on fandom as Wigan? Indeed, from what I can gather, achieving 1.5-3k fans allowed several clubs to score an A for fandom. 

There is no such thing as an A grade on fandom.

Wigan have more fans, higher utilisation, higher digital numbers and would ordinarily have higher viewers (it's mentioned there is an anomaly this year).

Their score would be different to London and Toulouse.

[Archie Gordon]

4 minutes ago, Dave T said:

I'm not sure that different approach is a problem. Why is a differing approach across two measures a bad thing?

Because if you have the data to allow for a continuous score, selecting thresholds adds an additional, unnecessary bias. Who decides the thresholds and why? Thresholds also incentivise odd behaviours - invest heavily to get from 2.75k to 3k fans, then stop. 

You are also throwing away useful data. If you know that club A gets 6k crowds and club B get 4.5k crowds, why actively choose not to differentiate?

I would be intrigued to be have been party to the discussions.

[Archie Gordon]

4 minutes ago, Dave T said:

I'm not sure that different approach is a problem. Why is a differing approach across two measures a bad thing?

Because if you have the data to allow for a continuous score, selecting thresholds adds an additional, unnecessary bias. Who decides the thresholds and why? Thresholds also incentivise odd behaviours - invest heavily to get from 2.75k to 3k fans, then stop. 

You are also throwing away useful data. If you know that club A gets 6k crowds and club B get 4.5k crowds, why actively choose not to differentiate?

I would be intrigued to be have been party to the discussions.

[Gooleboy]

8 hours ago, dboy said:

So when you posted "where was all this last year when it was already obvious these weaknesses in the criteria became clear?" ...you meant that YOU did point them out...but no-one else did?

Plenty of people saw plenty of holes in it.

And they said so. Including the clubs.

It was obvious from Day 1, that there would be changes as the grading system develops.

No-one, not even IMG/RFL, thought they had the criteria right in Year 1.

I don't get all the Hue and Cry, when traditional P&R would give you EXACTLY what we have for 2025.

 

Traditional P&R as you say would have given us the same outcome, but we knew the outcome before a ball was kicked. Also, before the GF, Toulouse knew they wouldn't be promoted even if they had won. So yes, same outcome but but very different situations to reach it.

[Archie Gordon]

5 minutes ago, Dave T said:

There is no such thing as an A grade on fandom.

Wigan have more fans, higher utilisation, higher digital numbers and would ordinarily have higher viewers (it's mentioned there is an anomaly this year).

Their score would be different to London and Toulouse.

I think TO scored 4.15/5 on fandom and Wigan scored 4.75/5. Both are well above 75%.

And this leads to a related point. 

You are of course right that you don't get an A for each pillar; that was me being a bit provocative. But my point is that it ought to be just as challenging to get 75% on fandom as it is to get 75% on community. Fandom is obviously not working as intended.

[dkw]

25 minutes ago, Archie Gordon said:

Both exercises have been categorically poor.

The survey was abysmal - dreadful questions, a self-selected sample, opportunity for one person to respond multiple times, etc.

For the grading, I can keep drip feeding examples but everyone knows them by now.

Here's another. Both TO and London scored A grades on fandom. They have very small fan bases indeed. Does anybody really think these two clubs are pretty much in the same space on fandom as Wigan? Indeed, from what I can gather, achieving 1.5-3k fans allowed several clubs to score an A for fandom. 

Again though, that's entirely based on your opinion, you have given absolutely no quantifiable data or outcomes to prove its failed.

[RP London]

18 minutes ago, Archie Gordon said:

I think TO scored 4.15/5 on fandom and Wigan scored 4.75/5. Both are well above 75%.

And this leads to a related point. 

You are of course right that you don't get an A for each pillar; that was me being a bit provocative. But my point is that it ought to be just as challenging to get 75% on fandom as it is to get 75% on community. Fandom is obviously not working as intended.

how was it intended to work? you've already said you would have loved to have been privy to the conversations around this, which means you weren't so how can you possibly say it "is obviously not working as intended" when you have no idea how it was intended to work. 

I am not a fan of some of the metrics but I don't know exactly what they were looking to get out of this. They may have wanted some specific areas to be be specifically covered right now and therefore put the marks in the sand there. knowing that clubs that already meet that can, indeed, not worry about any increase and concentrate elsewhere, and that that is actually ok. Therefore everyone is aiming for a base level in certain areas. 

As has been mentioned many many times over the 12 year period I think everyone would expect these criteria to change this is just the start and some clubs are very low in some important to IMG areas. Whether we like them or not, or even understand why they are there or not, we cannot possible say they are not doing what they intended unless we know exactly what they were intended to do and what IMG see as the next step and the next step. 

I still hold judgement on whether they have got this right. I dont see anything in the scoring that makes me go "wow I didnt expect the club to be in that position" except perhaps the number of A grades but those clubs would be up there anyway A or high B so little/no difference to me. The proof will be in how this develops now from here and what IMG do with these clubs and league that they see as being the future and have graded as such.

Edited October 30, 2024 by RP London

[Dave T]

21 minutes ago, Archie Gordon said:

Because if you have the data to allow for a continuous score, selecting thresholds adds an additional, unnecessary bias. Who decides the thresholds and why? Thresholds also incentivise odd behaviours - invest heavily to get from 2.75k to 3k fans, then stop. 

You are also throwing away useful data. If you know that club A gets 6k crowds and club B get 4.5k crowds, why actively choose not to differentiate?

I would be intrigued to be have been party to the discussions.

We need to be careful about stating things are wrong or poor because they may not be exactly how we would do it. Weightings need to be carefully considered when deciding the scoring and there is also the point about keeping things as simple as possible, while allowing for as much flexibility to move and improve.

It would be interesting to see every single metric set as a ranking system where possible, but then in reality this isn't necessarily a competition, it's about your own standards being the best they can.

 

[Archie Gordon]

There are things that everybody ought to agree are wrong. 

If you have a measure where every one of your 35 clubs scores the same - and there seem to be a few of these measures - that measure is just rubbish. That's not my opinion; it's how measurement works!

I didn't need to be in the meeting to know that.

[Dave T]

22 minutes ago, Archie Gordon said:

I think TO scored 4.15/5 on fandom and Wigan scored 4.75/5. Both are well above 75%.

And this leads to a related point. 

You are of course right that you don't get an A for each pillar; that was me being a bit provocative. But my point is that it ought to be just as challenging to get 75% on fandom as it is to get 75% on community. Fandom is obviously not working as intended.

The Fandom category is quite a generous one, but if you add in utilisation which could easily be factored in under here and it gives a better reflection of crowds etc. If there was an anomaly with the Sky figs this year, you'd also expect Wigan to.be a full 5 and Toulouse to remain around 4.15 (in fact drop lower as SL crowds roll off). So there is a clear differentiator there.

But if we added in utilisation, making it out of 6, then it would add even more of a differentiator.

[Dave T]

1 minute ago, Archie Gordon said:

There are things that everybody ought to agree are wrong. 

If you have a measure where every one of your 35 clubs scores the same - and there seem to be a few of these measures - that measure is just rubbish. That's not my opinion; it's how measurement works!

I didn't need to be in the meeting to know that.

Which ones?

[Dave T]

When we do anything like this, rather than over analysing things at a granular level, it's really important to look at the outcomes.

Has this driven the right outcomes? I don't think there is much controversy about the 12 teams selected. We could argue it's a touch generous to Cas and Wakey, but where they rank versus other clubs is probably fair.

So what would changing individual scoring metrics to some of those suggested do? Would it change the outcome?

It looks to me.that the outputs are pretty much exactly as you'd expect.

Where I think there could be a little focus is on how the rankings are used. At lower levels I think there is a challenge around benefit of this, maybe a light touch minimum standards and leave the full submissions to those with SL ambition is the way forward. That's how it worked under licensing I believe.

But if a system is delivering the right outcomes, maybe it isn't as wrong as people are making out.

[Archie Gordon]

I'll have a last effort to convince you.

Performance seems to me a decent indicator because it differentiates and provides a good range of scores - essentially 0 to 5, with a mean of about 2.

By contrast, Fandom is awful. It doesn't differentiate particularly well between a club with 15k fans and 200k SM followers from one that has 2k fans and 20k SM followers.

The mean score for Fandom seems much higher than for Performance, and there is too much clustering around this higher mean for it to properly differentiate the best clubs from the middling ones from the poorer ones.

None of this is my opinion. Once again, it gets to the fundamentals of measurement.

Edited October 30, 2024 by Archie Gordon

[Archie Gordon]

12 minutes ago, Dave T said:

When we do anything like this, rather than over analysing things at a granular level, it's really important to look at the outcomes.

Has this driven the right outcomes? I don't think there is much controversy about the 12 teams selected. We could argue it's a touch generous to Cas and Wakey, but where they rank versus other clubs is probably fair.

So what would changing individual scoring metrics to some of those suggested do? Would it change the outcome?

It looks to me.that the outputs are pretty much exactly as you'd expect.

Where I think there could be a little focus is on how the rankings are used. At lower levels I think there is a challenge around benefit of this, maybe a light touch minimum standards and leave the full submissions to those with SL ambition is the way forward. That's how it worked under licensing I believe.

But if a system is delivering the right outcomes, maybe it isn't as wrong as people are making out.

Precisely the same argument then for P&R. We have the same 12 clubs in SL for 2025.

[Archie Gordon]

1 hour ago, gingerjon said:

Assume for this purpose that you are in favour of gradings, and given this is the off season and the only alternative is work, what would your grading system look like?

I am in favour of gradings, yes.

Leaving aside my more granular problems with the gradings, I wouldn't have them count for so much. I would divvy out CF thus:

One portion of CF dependent on gradings; a second portion of CF dependent on which league you play in. Thus, clubs in the same league would get different amounts of CF; indeed, it might even be possible for a club in the Champ to get more CF than one in SL.

P&R remains but without the cliff edge. Incentives to improve your grading remain but without the cliff edge.

I think Barley Mow proposed something similar 18 months ago - apologies to him/her if I got that wrong. I haven't come across a better option since.

[Dave T]

25 minutes ago, Archie Gordon said:

Precisely the same argument then for P&R. We have the same 12 clubs in SL for 2025.

That does ignore a couple of things. Firstly, P&R saw London in SL and could easily see Fev or Bradford promoted who would not be selected via grading at the moment.

That's fine if people like that method, i get that, but they are delivering very different outcomes.

It also.ignores the outcomes where clubs have actually driven change i.e. facilities, financial strengthening and digital work. They are also outcomes that can be ignored under p&r.

[Barley Mow]

23 minutes ago, Archie Gordon said:

I am in favour of gradings, yes.

Leaving aside my more granular problems with the gradings, I wouldn't have them count for so much. I would divvy out CF thus:

One portion of CF dependent on gradings; a second portion of CF dependent on which league you play in. Thus, clubs in the same league would get different amounts of CF; indeed, it might even be possible for a club in the Champ to get more CF than one in SL.

P&R remains but without the cliff edge. Incentives to improve your grading remain but without the cliff edge.

I think Barley Mow proposed something similar 18 months ago - apologies to him/her if I got that wrong. I haven't come across a better option since.

As an advocate of on-field base P&R (subject to enforced minimum standards) my suggestion was that grading can be a good idea if the intention is to encourage clubs to improve off field operations, but I don't think it should be used to decide who is in SL.

I may have said that central funding could be linked to each club's score, and interestingly I believe they have now adopted that for funding in Championship and L1. I'm not entirely happy with that, as I don't think there should be a large difference in the amount of funding for different clubs in the same division, but I'm not sure what other 'reward' there can be for getting a good score in the grading other than a SL place and increased funding - Of those two options, the funding would be my preference. 

[Hopie]

1 hour ago, Dave T said:

When we do anything like this, rather than over analysing things at a granular level, it's really important to look at the outcomes.

Has this driven the right outcomes? I don't think there is much controversy about the 12 teams selected. We could argue it's a touch generous to Cas and Wakey, but where they rank versus other clubs is probably fair.

So what would changing individual scoring metrics to some of those suggested do? Would it change the outcome?

It looks to me.that the outputs are pretty much exactly as you'd expect.

Where I think there could be a little focus is on how the rankings are used. At lower levels I think there is a challenge around benefit of this, maybe a light touch minimum standards and leave the full submissions to those with SL ambition is the way forward. That's how it worked under licensing I believe.

But if a system is delivering the right outcomes, maybe it isn't as wrong as people are making out.

In general there is a division between those who support gradings and those who don't, and those who support gradings mostly either can't see or are happy to let problems with the system slide if it gives them the right vibe overall. 

But the principals of mathematics/formulae I have learned over the years are what I apply here. You can put your numbers into a formula and get the right outcome almost by accident, its only if the formula works for all combinations of numbers that it meets the test of time. Just because the teams promoted and relegated this year are the ones expected doesn't mean the scores work for all situations, and the issues rightly pointed out on this page, and those earlier in the thread should really have been addressed over a year ago at least.

[gingerjon]

44 minutes ago, Hopie said:

In general there is a division between those who support gradings and those who don't, and those who support gradings mostly either can't see or are happy to let problems with the system slide if it gives them the right vibe overall. 

But the principals of mathematics/formulae I have learned over the years are what I apply here. You can put your numbers into a formula and get the right outcome almost by accident, its only if the formula works for all combinations of numbers that it meets the test of time. Just because the teams promoted and relegated this year are the ones expected doesn't mean the scores work for all situations, and the issues rightly pointed out on this page, and those earlier in the thread should really have been addressed over a year ago at least.

What are those specific issues though?

I get attendances but that's because I think the steps get too broad and you get a big reward too early.

TV gantries, car park sizes etc ... those are good things to include and there will always be a cut off.

The big screen is a hilarious distraction - "The screen that a sport that doesn't use screens is fine for them so why should the little sport of rugby league want better" ...

Catchment is bobbins and should be replaced by the proper Community criterion that was meant to be included at the start.

Outside that ... what?

[RP London]

2 hours ago, Archie Gordon said:

There are things that everybody ought to agree are wrong. 

If you have a measure where every one of your 35 clubs scores the same - and there seem to be a few of these measures - that measure is just rubbish. That's not my opinion; it's how measurement works!

I didn't need to be in the meeting to know that.

Or that measurement worked perfectly becuase previously it was not met but now it is.. no matter how easy it was to accomplish people were not doing it before hand. However, it is an utterly vital thing for the rest of this process to fit together, therefore the measurement is good. I work with measurements too and we have some that are absolutely like this, easy to meet, should always be met and if they are not it is a big problem. You still measure it becuase it keeps the system honest and keeps peoples minds on the fact that that measurement MUST be met. 

This may not be the way you like it or are used to it but it doesn't mean it doesn't work it only doesnt work if it isnt doing what it is designed to do.

[RP London]

2 hours ago, Archie Gordon said:

I'll have a last effort to convince you.

Performance seems to me a decent indicator because it differentiates and provides a good range of scores - essentially 0 to 5, with a mean of about 2.

By contrast, Fandom is awful. It doesn't differentiate particularly well between a club with 15k fans and 200k SM followers from one that has 2k fans and 20k SM followers.

The mean score for Fandom seems much higher than for Performance, and there is too much clustering around this higher mean for it to properly differentiate the best clubs from the middling ones from the poorer ones.

None of this is my opinion. Once again, it gets to the fundamentals of measurement.

perhaps they are not looking at this as a pure measurement opportunity but also to use it as a target related opportunity. 

Those at the top end are seeing the benefit in other areas (and therefore this will be picked up in other areas of the grading). Those that were doing naff all are given a major incentive to get that up, by a huge percentage for them, to show them what these benefits are and to get them to a point where the next building block impacts on other grading levels. (SM and Fans etc can be monetised) but IMG maybe have worked out that if everyone gets to level x then they have something they can really work with, but if they dont get enough to x then "whats the point", after that you tweak it for future years to see growth or you expect that people will see how it benefits in other areas and just get on with it. 

My main gripe with this argument is that you dont know what this process' full aim was. Its not just to measure the clubs, thats clear, if you then dont know what it was you cannot judge it the way you are. There are multiple things they wanted, its clear to see that, because they have said the next steps in this phase (which people have laid out before) and this is just the starting point of being in a position to do that. 

you are being very pure in your terms of "measurement" but its not as simple as that IMHO.

[Archie Gordon]

8 minutes ago, RP London said:

Or that measurement worked perfectly becuase previously it was not met but now it is.. no matter how easy it was to accomplish people were not doing it before hand. However, it is an utterly vital thing for the rest of this process to fit together, therefore the measurement is good. I work with measurements too and we have some that are absolutely like this, easy to meet, should always be met and if they are not it is a big problem. You still measure it becuase it keeps the system honest and keeps peoples minds on the fact that that measurement MUST be met. 

This may not be the way you like it or are used to it but it doesn't mean it doesn't work it only doesnt work if it isnt doing what it is designed to do.

No. The tool for that is minimum standards.

[Hopie]

33 minutes ago, gingerjon said:

What are those specific issues though?

I get attendances but that's because I think the steps get too broad and you get a big reward too early.

TV gantries, car park sizes etc ... those are good things to include and there will always be a cut off.

The big screen is a hilarious distraction - "The screen that a sport that doesn't use screens is fine for them so why should the little sport of rugby league want better" ...

Catchment is bobbins and should be replaced by the proper Community criterion that was meant to be included at the start.

Outside that ... what?

Outside the 2 of the 5 criteria? (Fandom and Community ) Is that not enough?

Fandom is completely broken and any criteria that has things everybody gets, that are beyond the control of the clubs or that has poorly drawn lines in the sand (attendance is a great example) should have been written better in the first place and changed at any opportunity.

Of the other 3 I think performance is the best one, the stadium facilities is decent but could have a more graduated score, but I think the finance score is too easy to manipulate, I don't seriously think the finances of clubs have improved in the last year, so the score improvements show something else. Similar to the overall grading scores increasing and the number of As awarded, it shows more that the overall grading criteria were poorly chosen, rather than showing an improvement in clubs sustainability and growth.

 

Edit: One more broader point. For those saying you don't need to give points for x because that gives other points and IMG grading is just a broad exercise in improvement and not about specifics, why do we give bonus points for winning trophies? Would clubs not go for these trophies without bonus points? 

Edited October 30, 2024 by Hopie