The “Back to the Basics” series is designed to explore the foundation principals of statistical analysis across the four major American sports. The series will provide readers with an understanding of how teams approach roster construction and why certain decisions are made both on and off the field. Readers will also be directed to additional information sources, such as websites, books, or even magazine articles that could substantially increase their knowledge of the subject at hand.
Total Quarterback Rating [QBR]
Total Quarterback Rating [QBR] was developed by the ESPN Stats and Information Group as an alternative method for evaluating the NFL quarterback position. The idea was to create a metric that would numerically explain how a quarterback has contributed to scoring and winning, as opposed to merely measuring the four components of passing that comprise the NFL’s Passer Rating metric. However, in order to really understand QBR, we need to understand what ESPN was striving to “fix” about statistically evaluating quarterbacks. This post will examine a number of smaller topics, including: (1) Passer Rating; (2) the origins of QBR; (3) calculating QBR; and (4) the pros and cons of QBR.
Passer Rating: The Need for Change?
The NFL’s Passer Rating statistic has been under siege for years. The modern version of the statistic was developed in 1973 and remains as confusing to understand today as it was 20 or 30 years ago. It combines all of the basic statistics that are used to measure quarterback passing performance into a single figure, and it was designed as the ultimate evaluation tool.
The full formula for calculating Passer Rating and an exemplary calculation can be found at NFL.com, but we can highlight the important issues for this post. The four components of Passer Rating include: (1) Completion Percentage; (2) Average Yards Gained per Attempt; (3) Percentage of TD Passes per Attempt; and (4) Percentage of INT’s per attempt. The issues that critics have with Passer Rating are numerous, but mainly boil down to the following:
“Computing a statistic based off of flawed inputs does not make the resulting stat any less flawed.”
In essence, this is similar to a mechanic putting four flat tires on a car and then telling you that it is ok to drive. It just doesn’t make much sense. For example, completion percentage operates under the assumption that all passes are created equal, which they are not. A 30 yard completion in the air is far more difficult and requires a much higher degree of skill than a screen pass behind the line of scrimmage, yet Completion Percentage treats each pass the same. Furthermore, if the running back that catches that screen pass spins, jukes and bulldozes his way for a 30 yard gain, the quarterback is credited with the exact same Average Yards per Attempt and the exact same Passer Rating as well.
Interception Percentage is also a flawed input when calculating Passer Rating. For example, Interception Percentage does not take into account the difference between a bad throw and an interception. If a receiver bobbles a catch and, as a result, it is intercepted by a defender, the play is still counted as an interception for the quarterback and negatively impacts his rating (even if the ball was perfectly delivered). This is also true when a ball is tipped, a receiver falls down, etc. Furthermore, Nathan Jahnke from Pro Football Focus points out that, much like completion percentage, game situations factor into interception rates. For example, Jahnke points out that interceptions occur at a far greater rate as passes are thrown further and further downfield, and that when pressured, quarterbacks are 50% more likely to throw an interception. Therefore, poor line play, game situations, and offensive schemes play a substantial role in determining the likelihood of a quarterback throwing an interception, yet interception rate fails to account for all of these factors.
I have yet to find a columnist who vigorously opposes Touchdown Percentage as an NFL statistic (but I’m sure there is someone out there and I just have not found him to date). My own issue with the statistic is that it de-values the high usage quarterback such as Matt Ryan in favor of quarterbacks like Cam Newton who, though efficient, do not have their arms relied on as much by their teams. For example, in the 2013-14 season, Newton’s 5.07 TD% is far superior compared to Ryan’s 3.99%, yet the Falcons relied on Ryan’s arm to get them yards almost 38% times more often than the Panthers relied on Newton. Furthermore, the lack of a running game in Atlanta [ranked 32nd] likely forced Ryan to take more chances throwing downfield in order to generate more offensive yardage and obtain first downs. In contrast, the Panthers sported one of the better rushing attacks in the NFL [ranked 11th], likely allowing for more favorable passing situations for Cam.
After hearing all of that, you might be wondering why the NFL still keeps passer rating around. It may come as a surprise, but the stat still has a number of advocates, and if you’d like to read their takes on passer rating, I have found this post from Cold Hard Football Facts to be the most vigorous and well-reasoned defense.
The origins of QBR
One of the main critiques of ESPN’s Total QBR statistic is that it is proprietary in nature. ESPN owns the statistic, and currently [to my knowledge] has not licensed it to other outlets such as NFL.com, CBSSports.com, etc. This is why ESPN is the only place you can find QBR. It drives users to their site and increases the exposure of ESPN as a product. However, my answer to this critique is this:
The guys who developed this stat for ESPN are who we in Boston call “wicked smaht,” and did not need to develop this statistic to further their careers.
ESPN put together their QBR team by poaching some of the best talent in the Sports Analytics community. Jeff Bennett, the Senior Director of Production Analytics and Research at ESPN, put a team together where Dean Oliver was hired to spearhead the research on the concept and theories of QBR, and his team included the likes of Alok Pattani, Albert Larcada, and Professor Ben Alamar. The inclusion of Oliver and Alamar alone should put some weight behind this research, as both are considered outstanding sports analytics professionals and have authored some of the best works in sports analytics to date. [Oliver is the author of “Basketball on Paper” and Alamar is the author of “Sports Analytics: A Guide for Coaches, Managers, and Other Decision Makers and the founding editor of the Journal of Quantitative Analysis in Sport”] A brief summary of each team member’s contributions to the development of QBR can be found here.
The point is that to say that this statistic was created merely to drive ratings to ESPN is ludicrous. It has been a viable step in the right direction of properly evaluating quarterback play in the NFL and will be around for a long time.
The goal for ESPN's team was to gain a better understanding of the value a quarterback generates within each play, whether the play is made via his arm, feet, or brain. In sum, what did the quarterback do to contribute to his team’s chances of winning on each play? In the next sections we will examine exactly what QBR is and the theory behind it.
No Accessible Formula
Ok, here’s the deal. This post is not going to inform you on how to calculate ESPN’s Total Quarterback Rating [QBR]. As mentioned earlier, ESPN owns the QBR statistic, and has kept its exact formula’s out of the reach of the public. However, even though we cannot tell you exactly how to calculate QBR, we can show you the process and theory behind its calculation. In Oliver’s own explanation, he cites the QBR process as followed:
(1) Determine the +/- effect a play has on a drive's Expected Points total
(2) Divide the credit for that +/- effect between offensive teammates involved in the play
(3) Determine how “clutch” the play situation is relative to the outcome of the game
(4) Convert the resulting rating to a 0-100 scale
(1) Expected Points
The concept of expected points is relatively simple to understand. The basic concept is that, as you advance closer you are to the opponent’s end zone, the likelihood of your drive ending in a touchdown or field goal increases. Thus, while an offense advances down the field, it is accumulating “point potential”, a concept that was first discussed in the 1980’s in the book “The Hidden Game of Football.” For example, a situation of 1st and 10 from the opponent's 40 has more point potential than 1st and 10 from your own 40 yard line.
It is important to make note that Expected Points really means Expected Net Points, because the value can be negative. This merely means that given the situation, it is likely that the opponent will score. For example, a 4th and 10 situation from your own 5 yard line will likely have a negative expected point value. Why? After punting, the opposing team will have the ball in a situation where they are more likely to score than you are on that 4th and 10 play.
Brian Burke from AdvancedNFLStats.com took this concept one step further by factoring in downs, noting that a 3rd and 10 from the opponent's 40 has less expected value than 1st and 10 from the opponent's 40. Oliver and his team has gone even further by incorporating the following: (1) Clock Time; (2) Home Field; (3) Timeouts; and even (4) Field Surface. QBR measures the expected points added during a play, and determines this by utilizing the following formula [basic version]:
Expected Points Added = [Expected Points after Play] – [Expected Points prior to Play]
(2) Dividing Credit
Once the Expected Points Added has been established, the team goes about dividing up the credit for the positive or negative points. This division of credit goes to every member of the team, including: (1) The Offensive Line; (2) Running Backs; (3) Receivers; and (4) Quarterback. Oliver provides us with the following example:
“On a pass play, for instance, there are a few basic components: (1) The pass protection; (2) The throw; (3) The catch; (4) The run after the catch.
In the first segment, the blockers and the quarterback have responsibility for keeping the play alive, and the receivers have to get open for a QB to avoid a sack or having to throw the ball away. On the throw itself, a quarterback has to throw an accurate ball to the intended receiver. Certain receivers might run better or worse routes, so the ability of a QB to be on target also relates somewhat to the receivers. For the catch, it might be a very easy one where the QB laid it in right in stride and no defenders were there to distract the receiver. Or it could be that the QB threaded a needle and defenders absolutely hammered the receiver as he caught the ball, making it difficult to hold on. So even the catch is about both the receiver and the QB. Finally, the run after the catch depends on whether a QB hit the receiver in stride beyond the defense and on the ability of a receiver to be elusive. Whatever credit we give to the blockers, receivers and quarterback in these situations is designed to sum to the team expected points added.”
Dean Oliver, ESPN.com
The team utilizes ESPN video tracking technology to analyze and assign credit or blame between the QB and the receiver on these plays. However, lack of access to the video technology, not to mention the team’s analytical procedures for dividing credit, makes it extremely difficult to speculate on the formulas for this step in a QBR computation. The Expected Points Added that is credited to the QB is used in QBR computations. It is important to note that credit is given for extending plays via scrambling and pocket presence, allowing for all actions taken by the quarterback to be accounted for and not merely a pass that he throws.
(3) Clutch Index
The next factor that goes into QBR is the Clutch Factor. Clutch Factor analyzes the game situation and increases the Expected Value Added on a play by the importance of the situation. For example, 1st and Goal from the opponent's 10 yard line at the start of the second quarter of a tie game has a clutch index of 1.0 [a normal play], but the same scenario is given a higher Clutch Factor if it is occurring in the 4th quarter. Why? The lack of time remaining in the game significantly attributes to the likelihood of the team winning should they succeed in getting a touchdown or field goal.
Clutch Factors are determined prior to every play. Oliver has indicated that Clutch Factors [indices] range from .3 to 3.0 depending on the game situation. This is the part of QBR which rewards QBs who perform at a high level in the fourth quarter and come from behind situations [See Andrew Luck rookie season v. second season].
(4) Convert to 0-100 Scale
The final step before presenting QBR totals is to scale it to the 0-100 range. Oliver has indicated that this is simply done via a mathematical formula and has little significance other than to make the stat easier to interpret. The reasoning being that a 0-100 scale, with 50 being considered average is more easily understood than the 158.3 perfect passer rating.
Pros / Cons of QBR
The goal of this post was to lay the information available on QBR in front of you and allow you to make your own conclusions about it. However, during this process I have found a couple of things that have routinely been mentioned in critiquing ESPN’s statistic.
- QBR awards more value to plays occurring in the second half than in the first, even though plays in the first half are significant in setting up the second half situations.
- QBR attributes too many points to running quarterbacks [i.e. Tim Tebow]
- QBR does not account for everything a QB does prior to a play [i.e. Peyton Manning’s ability to check the offense into more favorable play calls]
- QBR does not adjust for defensive strength.
These are some of the more frequent ones I have found to date. I’m sure there are more out there, but these all seem to be missing a valuable point. This statistic is, at worst, a step in the right direction for analyzing quarterback play in the NFL. It is right up there with the work being conducted at FootballOutsiders.com and AdvancedNFLStats.com as a leading statistical breakthrough in this analytical field.
As for the critiques, I tend to agree with the first and second ones. In no situation should Tim Tebow be awarded a higher QBR just because he decided to finally start making plays in the 4th quarter, especially those he had to make scrambling because he lacked the arm talent to throw downfield [full disclosure, I love the guy as a person]. As for the third and fourth critiques, Oliver has mentioned that they tried to find a way to incorporate pre-snap play into QBR, but that it is “nearly impossible” to do so. Furthermore, the stat was purposefully left un-adjusted for defensive strength as the team wanted the statistic to be as flexible as possible, thereby allowing users to sort for a quarterback's QBR in certain situations such as “3rd and longs” or “Plays in opponent territory,” etc. However, this does not mean that it cannot be adjusted if an analysis calls for it.
Total Quarterback Rating [QBR] was designed and developed by ESPN in order to utilize the vast amounts of data they have and convert it into a proprietary statistical measure. Although the public currently does not have access to ESPN’s data or method of calculating the statistic, we have to appreciate the method and research that went into its formation. As of now, all that we have available to us is a description of the process. If you would like a more in-depth breakdown on QBR, I invite you to check out Bennett and his team’s own presentation on QBR from the 2012 MIT Sloan Sports Analytics Conference.
Related Info: Passer Rating; DVOA
You can read other Back to the Basics articles here.
Jeff Meehan is a 2nd year JD/MBA candidate at Suffolk University and recently interned in the Boston Red Sox legal department.