Turning Words Into Touchdowns

By : May 2, 2011

Does a player’s speech predict how he’ll perform in the NFL?
By Michael Agger | www.slate.com

Imagine you’re Carolina Panthers general manager Marty Hurney. Your team went 2-14 last year. You have the first pick in the NFL draft on Thursday. All of the pundits are predicting that you will take Cam Newton, the explosive Auburn quarterback who won the Heisman Trophy and led his team to the national championship. You absolutely need a quarterback to be the cornerstone of your franchise. Is Cam your guy?

You have his college stats. You have his highlight reel. You watched him work out at the NFL combine. You have his score on the Wonderlic—the controversial intelligence test that’s been administered at the combine since the 1970s. Your coaches have interviewed him. Your scouts have timed him. You’ve asked around about his character, his on-field demeanor, his football smarts. What’s left to consider?

I’ll pull out some of the scintillating highlights:
I’m just blessed to be in the situation and being able to make plays when it was time for me to make plays.
We’re always happy with a win.
I’m happy with the accolade, but I feel that I didn’t play the best game I could have played.

Could it really be true that these innocuous statements can help assess Cam Newton’s pro potential? That’s the assertion, mission, and business plan of an Ohio-based company called Achievement Metrics. It analyzes the speech of star college players, looking for traits such as “conceptual complexity,” “need for power,” and “deliberativeness.” It compares similar players and correlates these traits with future performance. College wide receivers whose speech shows low levels of distrust, for example, have a greater probability of becoming Pro Bowlers than their less-trusting counterparts.

That may sound batty, but it’s closely related to established social science. For decades, political scientists have counted the words and analyzed the grammar of political speeches in order to understand a leader’s state of mind. In recent years, this process has become both automated and refined by algorithms and data crunching. Achievement Metrics has grown out of a company called Social Science Automation that does most of its work for the U.S. government. It is attempting to bring text analysis to nontraditional areas, like the CEO’s corner office and, yes, the scouting of pro athletes.

The analysts at SSA don’t say a lot publicly, as much of what they do is classified. You can get a flavor of their work, though, in a 2005 paper titled “The Distinctive Language of Terrorists.” The paper discusses the technique of remote assessment, which involves gathering samples of speech and determining whether someone is likely to be a terrorist. The paper demonstrates the validity of remote assessment by analyzing the speech of political leaders and known terrorists and scoring them in such categories as self-confidence, task orientation, and distrust of others. Remote assessment sorted out the bad guys from the good with impressive accuracy—although, like all probabilistic models, it was not perfect. John Kerry, for example, was classified as a terrorist. (The speech that the study sampled came from the months in the 2004 campaign when he went negative.)

The speech cues that SSA considers important are not the obvious ones—”I want to kill all Americans!”—but rather instances of words like I and we, as well as the use of qualifiers—I maybe think I want to kill some Americans, perhaps. While that’s a crude presentation of the work, you get the idea: Subtle, almost undetectable patterns can be analyzed to determine how likely a political figure is to engage in terrorist activity. Roger Hall, a consulting psychologist for SSA and CEO of Achievement Metrics, claims, “If you give us unidentified speech text, we can distinguish terrorists with 90 percent accuracy.” Now you understand why they don’t talk publicly about their work. Is the leader of that new group in Afghanistan blowing hot air or is he trouble? That’s the kind of question the intelligence community is interested in.

Back to football. About five years ago, SSA analyst Steven Hofmann and a colleague started thinking about how their work could be applied to the NFL. At the time, the league was having trouble with a rash of arrests and suspensions, and coaches and scouts wanted to know who they could count on to stay on the right side of the law. Hofmann began collecting interviews with college stars—he needs only about two pages of text to do his analysis. He produced this chart:

Players whose language displays both a lack of self-confidence and a high degree of self-centeredness presented a greater risk of being arrested or suspended. For players in the upper-right quadrant, the risk was estimated at 30 percent. Again, this chart deals in probabilities. But if a team is making a $20 million investment in a player, it’s useful to know if he shares common traits with players who have a tendency to get in trouble.

As with terrorists, there are no blatant words that signify whether a player is going to be a con. The analysis does not consider regionalisms or racial inflections—in fact, most of that kind of speech is corrected in the transcripts that Achievement Metrics sifts through. These are the kinds of quotes under the microscope: “I tweaked my knee in the third quarter” or “I should have cut across the field earlier.” And here are some of the words that the algorithm sifted and analyzed to produce its result: nice, hard, short, cornerbacks, talent, comfortable, catch.

What this chart does ask you to do is believe that spontaneous speech reflects our character. On one level, this is common sense, since we judge people by the words they use every day. Yet Achievement Metrics hasn’t stopped there. Along with using language to predict off-field outcomes, it’s also trying to parse speech for clues about on-field performance. Language, then, not only reflects our character. It reflects our potential—that future backup quarterbacks talk like backups, and future starters talk like starters.

Analyst Steven Hofmann brings up the example of the 2005 draft. The two top quarterback prospects that year were Alex Smith and Aaron Rodgers. Smith played for the University of Utah and wowed the scouts with his athletic ability at the NFL combine. He also scored an amazing 40 out of 50 on the Wonderlic intelligence test. The 49ers, who saw him as a smart, gifted athlete, made him the first pick in the draft. Rodgers, who many thought would go No. 1, had the painful experience of lingering in the green room on live television until he was selected by the Packers as the 24th pick.

Hofmann, as we know, has been gathering the college press conferences of first-round draft picks and analyzing them. Here is one of the charts on quarterbacks:

The y axis is the positive power score—the belief in one’s ability to influence events and outcomes: “I refuse to consider that” or “I promise that it won’t happen again.” The x axis shows the in-group affiliation score, terms and expressions that indicate whether the speaker associates positively with a group: “We did a great job today” or “Our linebackers were amazing.” These two traits are correlated with a player’s career NFL passer rating.

What would you have done if you were the 49ers general manager on draft day in 2005? The guy you like, Alex Smith, has low scores in positive power and in-group affiliation. He talks like someone who doesn’t see himself as a leader. The guy you are about to pass over, Aaron Rodgers, is at the opposite end of the spectrum. He’s a guy who talks like Julius Caesar. My guess: You probably would have dismissed it as academic hooey.

Achievement Metrics is trying to persuade NFL executives that it’s not selling pointy-headed gibberish. It’s presented its work at the premier sports stats conference at MIT and has been approached by curious teams. At the very least, this speech research seems more persuasive than the Wonderlic, which has been shown in repeated studies to have no ability to predict future performance. One reason the Wonderlic remains despite its uselessness is that we can’t measure the so-called “intangibles.” The private interviews the players have with teams, the two-hour psychological exam that the Giants used to administer, the personality tests—all of these are about trying to gain some purchase on what separates a Tom Brady from a Brady Quinn.

The beauty of what Achievement Metrics is trying to do is that spontaneous speech cannot be faked. (This is in contrast to the Wonderlic, which you can prep for like the SAT.) Textual analysis is not a crystal ball, of course. But look again at the chart above. There are six quarterbacks whose positive power and in-group affiliation scores were both above the median: Aaron Rodgers, Philip Rivers, Jay Cutler, Josh Freeman, Jason Campbell, and Rex Grossman. There are also six signal-callers whose positive power and in-group affiliation scores were both below the median: Alex Smith, Brady Quinn, Matt Leinart, Vince Young, Matthew Stafford, and Carson Palmer. The former group has thrown 470 career touchdown passes. The latter has thrown 290, with more than half of those belonging to Palmer. If I were an NFL general manager, I’d feel a lot more comfortable if the guy I was drafting was in that first group.

Achievement Metrics’ Roger Hall and Steven Hofmann were upfront about the fact that they deal in probabilities, not certainties. “If all the correlations were perfect,” Hall told me, “We would be lying.” Even so, they are refining their analyses and gaining more confidence. This year, they told me, they see a similar pattern to 2005. There are five supposedly solid prospects in this year’s quarterback class—Cam Newton, Blaine Gabbert, Jake Locker, Ryan Mallett, and Andy Dalton. One of these men, they told me, is in the quadrant of players who have not gone on to great success in the NFL.

Hall and Hofmann would not identify which quarterback is in the quadrant of doom. They are keeping this year’s data proprietary in the hopes that NFL teams will pay them for their research. If you were the Carolina Panthers, how much would you be willing to spend to find out whether Cam Newton scores like Alex Smith or Aaron Rodgers?

Filed Under: News

Leave a Reply

You must be logged in to post a comment.