Friday, May 9, 2014

THE PGSC SCORE, PART I: PGSC (PERFORMANCE, SONG FRESHNESS, GROWTH AND CONSISTENCY) VERSUS ACTUAL IDOL RANK



So I recently created a model which I thought was able to perfectly encapsulate the general fan's assessments of the majority top ten American Idol contestants, starting from AI3 all the way to what we have now in AI13, independent of producer and judge manipulation. We start with AI3 because that's when we are able to chart one of our metrics, as will be explained below. This is the likely ranking of the contestant had every contestant played on a neutral ground.




I've been able to scour whatnottosing's databases for this project, utilizing four core metrics which I believe make up the general fan's opinion on the contestants. Three of these metrics will be given equal weighting (out of a 100, upon within-season normalization), while the fourth measures the consistency of values between these three measurements. Remember, this normalization is done WITHIN the contestant's respective season. It is not normalized across all Idol seasons, because I want to capture how the contestant was like within their season, relative to their competition. Without further ado, the metrics are:

1) Song freshness factor. This is my personal favorite metric, based off the song repeatability factor on Idol, the song age, and its freshness.
2) Contestant growth. WNTS uses a linear regression across contestant performances over the course of the season.
3) Contestant performance. Usually the standard to determine the value of the contestant, but just like you can't use points per game to determine the value of an NBA player, we have to consider other metrics. There's a huge reason why the best singers often do not win the show any given year.
4) Deviation between the first three. This acts as a further normalizer, so that one contestant is not gaining the majority of their points in one category while being very weak in other categories (ahem, Tim Urban, Kristy Lee Cook, etc).

So the first three metrics are the ones given equal weighting, and the deviation subtracts off the normalized summation of those weights. We come up with a consolidate value which I think attempts to capture the value of the contestant, which we take the first initials of performance, song freshness, growth and consistency: or the PGSC score. Due to the song freshness and growth weighting (growth is why we start at the top 10, because we get enough performances to measure that mark), it might also give some value into the marketability of the contestant. Several trends emerge: a contestant that has a high consolidate ranking tends to be more marketable than one that does not. Usually, this mark is above 200 or above for the very high ranking, and there are several contestants who have breached that mark. An exception can the highest ranking contestant of that particular season. Obviously, no model is perfect and there are exceptions.

But here are some food-for-thought questions that might be addressed by the model (scroll your cursor over the data points to learn who the data point corresponds to, their respective scores for the four metrics, their total score, and other extraneous data):

1) For AI12: In an even playing field, would Candice Glover have won over Kree Harrison or Angie Miller? Candice delivered great vocal performances, but was sorely lacking in song freshness factor with respect to her season, but she was majorly pimped. There is an argument to made for Amber Holcomb making it over Burnell Taylor as well. In general, TPTB did their job that season: they wanted a girl to win so badly they casted an all-around awful group of guys who would struggle in all four core facets of the algorithm, and right on cue, they all went down easily.

2) The double elimination conundrum, for AI6 and AI8: there was a lot made that producers really wanted to get rid of AI6's Phil Stacey and AI8's Anoop Desai to make way for who they wanted as their true frontrunners. Both Stacey and Desai had consolidate scores well north of 200, and there's an argument to be made that had Melinda Doolittle and Jordin Sparks, or Danny Gokey and Adam Lambert not been so pimped in those respective seasons, both those guys would have went farther than 5th-6th place.

3) AI7 was a weird case, with Michael Johns--he wasn't exactly depimped, but he had such a high score. Which goes to show you--a very high score, as we'll see below, has repercussions.

4) In general...diva belters have been docked off since Jennifer Hudson and LaToya Hudson ranked extremely well in AI3. Largely, this is due to old, un-fresh song choices; but if you wondered why Anwar Robinson, Mandisa, LaKisha Jones, Lil Rounds, Jacob Lusk, Joshua Ledet, and faux belters like Pia Toscano and Jessica Sanchez ranked awfully (and why they were both eliminated early; until Jessica got saved)--that's your answer. Consider all parts of the algorithm here.

Another issue I wanted to explore was developing a regression curve between genders: for a given PGSC score, how does the audience react to particular gender? For those that are stat-geeks, I've used the p-test to compare the correlation between the genders, and these groups were determined to be distinct. The r-squared value is about 41%, so there is a lot of noise, but that's to be expected when you are dealing with so many data points (between both genders, there are 110 total). To ensure the highest r-test value, I've decided to use an 8-degree polynomial model (8 is the highest it can go to). There are several things to note:

1) A PGSC score within 160-190 is the proverbial sweet spot for a male singer. Every male within this range has made it to at least the top five, with winners David Cook, Scotty McCreery and Phillip Phillips coming out of it with scores in that range. The regression curve reflects that as the sweet spot as well. With scores of a 168 and 171, respectively, for Caleb Johnson and Alex Preston, those two aren't going to buck that trend, because they are both in the top three. Within this range, it's been a foolproof method for a top five contestant.  Females tend to have it rougher in this range, but to be fair it's mostly due to the early seasons with shock eliminations (Amy Adams, Jennifer Hudson).

2) It is VIRTUALLY IMPOSSIBLE to win with a PGSC score of 200 or above. There's a saying that you are too good for your own good...well, the contestant is TOO good for the audience's good, in this case. Males especially have it rough with a score this high, but there is ONE major exception: Kris Allen. But AI8 was a weird season: that season had THREE contestants with scores over 200 or above: Adam Lambert, Anoop Desai and Allen. And Allen was third in the pecking order of those three in score, so while he was very good, there were contestants that season who were better. So he wasn't exactly "too good for his own good" relatively speaking. Of the guys, there were also shocking eliminations with Phil Stacey and Michael Johns here too. The girls also produced no winners here, but had several in the range of 2-4: LaToya London, Kree Harrison, Haley Reinhart and Angie Miller. But again: too good for the audience's good.

3) It's hard to make a great argument that the audience votes for competent guys more than competent girls. According to the regression model, there are a few bumps here and there, but up to a PGSC score of about 160, the regression lines between males and females both oscillate and seemingly alternate in favoritism. The biggest issue is when we actually get to the very competent girls: at scores between 160 to 190, the girls get shafted in voting rank compared to their male counterparts. Like every model, there are key exceptions: in fact, our four female winners had PGSC scores between 140 and 170; it's just that there are many other female data points around that range that brings the regression curve downwards. In general, the regression argues that a girl's best chance to get far (but not win) is to get a score over 200. But ultimately, the best chance to win is a score in that 140-170 range for a girl. Jena Irene is right at that range, and she's in the top three. She has a good chance of winning, given that there is precedent.

This season: So yep, we have already spoiled it: Jena Irene, Alex Preston and Caleb Johnson (our top three) are locked in a dead heat. Jena's PGSC is 173, Alex's is 171, and Caleb is 168. (Hell, Jessica Meuse's is 169, making this is the most natural-entropy-guided top four in recent memory; in other words, this is the correct top four, based on PGSC scores, in spite of producer manipulation). All three of those scores are locked within a bracket that can produce winners for a male or a female, so this is literally anyone's game, especially given how tight the PGSC scores are, especially compared to past seasons. In the past, I can use the scores to make key predictions on who would win with a certain accuracy, but it's hard to do exactly that this season. It's a virtual toss-up.

Also, http://public.tableausoftware.com/views/idol_score_rank/Sheet1?:showVizHome=no#3  is the link to use, if you want to play with the larger model than what is shown below.

If you're not into visual data and you need the rankings spelled out for you, they are listed below. Below is the listing, or "camp-should-have-been", if you want to use that term, of the contestants, the way they would have ranked had TPTB or producers not interfered or done any sort of pre and within-show pimping, and let the contestant's performances speak for themselves. Also listed are the average PGSCs of the contestants within the top 10, and that can represent the ratings flow of the particular season (but not necessarily quality), based on those four factors. Without further ado, the PGSCs:

AI3:
1) LaToya London 223
2) Jennifer Hudson 180
3) Amy Adams 176
4) Fantasia Barrino 169
5) Diana DeGarmo 166
6) George Huff 151
7) Jon Peter Lewis 98
8) John Stevens 60
9) Jasmine Trias 59
10) Camile Velasco 19
AVERAGE: 130.1

AI4:
1) Carrie Underwood 170
2) Bo Bice 159
3) Nikko Smith 151
4) Constantine Maroulis 147
5) Anthony Fedorov 122
6) Nadia Turner 121
7) Jessica Sierra 118
8) Vonzell Solomon 111
9) Anwar Robinson 58
10) Scott Savol 52
AVERAGE: 120.9

AI5:
1) Chris Daughtry 200
2) Bucky Covington 157
3) Taylor Hicks 148
4) Paris Bennett 148
5) Elliott Yamin 144
6) Mandisa 121
7) Katharine McPhee 106
8) Ace Young 102
9) Kellie Pickler 57
10) Lisa Tucker 54
AVERAGE: 123.7

AI6:
1) Phil Stacey 213
2) Blake Lewis 192
3) Jordin Sparks 147
4) Melinda Doolittle 142
5) Gina Glocksen 135
6) Chris Richardson 127
7) Chris Sligh 88
8) Haley Scarnato 81
9) LaKisha Jones 76
10) Sanjaya Malakar 58
AVERAGE: 125.9

AI7:
1) Michael Johns 230
2) David Cook 187
3) Kristy Lee Cook 143
4) David Archuleta 130
5) Chikezie 129
6) Syesha Mercado 124
7) Carly Smithson 123
8) Brooke White 92
9) Jason Castro 68
10) Ramiele Malubay 28
AVERAGE: 125.4

AI8:
1) Adam Lambert 240
2) Anoop Desai 224
3) Kris Allen 216
4) Matt Giraud 166
5) Allison Iraheta 155
6) Danny Gokey 110
7) Scott MacIntyre 81
8) Lil Rounds 53
9) Megan Joy 42
10) Michael Sarver 35
AVERAGE: 132.2

AI9:
1) Lee DeWyze 190
2) Casey James 155
3) Crystal Bowersox 154
4) Michael Lynche 123
5) Tim Urban 109
6) Katie Stevens 106
7) Didi Benami 91
8) Aaron Kelly 73
9) Siobhan Magnus 73
10) Andrew Garcia 72
AVERAGE: 114.6

AI10:
1) James Durbin 214
2) Haley Reinhart 209
3) Lauren Alaina 188
4) Scotty McCreery 176
5) Casey Abrams 139
6) Pia Toscano 113
7) Stefano Langone 94
8) Paul McDonald 82
9) Jacob Lusk 46
10) Naima Adedapo 32
AVERAGE: 129.3

AI11:
1) Phillip Phillips 167
2) Hollie Cavanagh 165
3) Skylar Laine 136
4) Elise Testone 129
5) Erika Van Pelt 124
6) DeAndre Brackensick 118
7) Colton Dixon 107
8) Joshua Ledet 107
9) Jessica Sanchez 106
10) Heejun Han 52
AVERAGE: 121.1

AI12: (note: Curtis Finch had no growth rate, so his data was omitted; not that he would have rated well, anyway):
1) Kree Harrison 231
2) Angie Miller 230
3) Janelle Arthur 185
4) Candice Glover 157
5) Burnell Taylor 139
6) Amber Holcomb 111
7) Devin Velez 100
8) Paul Jolley 80
9) Lazaro Arbos 51

And where we are at now...AI13 so far:

1) Jena Irene 173
2) Alex Preston 171
3) Jessica Meuse 169
4) Caleb Johnson 168
5) Majesty Rose 124
6) Malaya Watson 112
7) Sam Woolf 106
8) Dexter Roberts 98
9) CJ Harris 74
10) MK Nobilette 40
AVERAGE: 123.5

Now, below, play with the actual model!:



































No comments:

Post a Comment