Futures: Scouting's Bum Rap

by Matt Waldman

Scouting gets a bum rap.

"Of course Waldman would say this," you proclaim. "He’s a scout!"

I may perform the fundamental role of one, but I am not a scout. This elicits laughter from my friend Ryan Riddle. The Bleacher Report columnist who holds Cal’s single season sack record and played with the Raiders, Ravens, and Jets says I have a misplaced sense of honor when it comes to refusing to wear that label.

I prefer talent evaluator, tape watcher, tapehound, or tapehead. My friends –- if I have any left since I started doing this work eight years ago –- might say "Film Hermit" is the best fit. I’ve never worked for an NFL team, so these names seem more suitable to me. Scouts have responsibilities that I don’t –- among them is reporting to management within a company structure.

If you have the chance to learn about the pre-draft process for most NFL teams, scouting is the study of a player’s positive and negative characteristics. It’s also an evaluation of how easy it is to fix the player’s issues and his potential fit within a team system. But based on what former scouts, coaches, and general managers of NFL teams say about the machinations that go into a team’s draft, I am thankful that I am not a scout.

While fans and writers may take the lazy route and blame picks gone wrong on poor scouting, it’s the general manager, coach, and owner who hold the weight of the decision-making power. This is a huge reason why scouting gets a bum rap.

To take it a step further, I’ll advance the popular Bill Parcells analogy of "buying the groceries." I can spend months in the grocery store and tell you that it has quality cuts of grass-fed steak; a delicious, rosemary batard baked in-house; and every variety of apple found in North America. But if those holding the wallet or cooking the food demand a papaya, I can tell them until I’m blue in the face that if they want a good one, it’s only found in Jamaica and they’re still going to pick an unripe one, take it home, prepare it, and then watch it spoil the meal.

It doesn’t help matters when I have to read Mike Tanier describe draft analysis as a pseudoscience. He’s right for the wrong reasons. Scouting is a craft, not a science. However, teams haven’t made it the same priority to address opportunities to improve scouting the way they have upgraded technology and embraced other forms of analysis.

With all the advances that the NFL has made with equipment, strategy, cap management, and technology, they haven’t done enough to advance the process of talent evaluation. It shouldn’t be the sports equivalent of Madam Zora’s, but until teams address the problems, Tanier gets to write entertaining draft pieces at their expense.

I think there is a lot that teams can do to improve their talent evaluation processes. What I will propose here are things I’ve learned from my experience in operations and process improvement. I base my solutions on problems I’ve gleaned in conversations with former scouts, reading and listening to former NFL general managers talk about their past roles, and extensive study of college prospects for the past eight years.

Some of these ideas may be new to the NFL, but I don’t begin to think they are revolutionary in the scope of other industries. I’m sharing these things because it’s too easy to listen to a gray-haired man in a suit on a television network and take what he says as gospel –- processes that may, in fact, be fundamentally flawed -- and then see that gospel perpetuated from generation to generation of football men.

When viewing NFL front offices and how they cope with change, I get the impression that many of them have a buttoned-up, low-risk culture similar in dynamic to Wall Street. It takes a lot for newer ideas to take hold in an NFL front office, as it does for an investment bank to accept "new blood" from a business school lacking a history of established connections with the firm as a personnel pipeline.

Some of what I’ll suggest is not even about new ideas; just better implementation of old concepts. The first point below is a good example where leaders tend to talk the talk better than they walk it.

Develop a Common Vision

Former NFL GM Ted Sundquist said on Twitter this week that there is a natural difference in preference when it comes to what personnel staff and coaches value in a prospect. Coaches gravitate to players with refined skills who are ready to play now. Personnel men and scouts are more forgiving of great athletes who can be "coached up."

The frame of reference each role has with its job security is a factor with these differences in views: coaches know they need to win now. The downside of acquiring a raw, athletic prospect is that this player may never learn at the pace to have a positive impact. If that prospect does figure it out, that coach knows it may not happen within a time frame that will help him keep his job.

Moreover, the new coach has now inherited a team with players that either don’t fit his system or haven’t demonstrated what it takes to produce on the field. Salary cap considerations become a factor in getting rid of that player.

Talk talent over need all you want, but what I’m really talking about is differentiating between usable talent and useless talent. If general personnel men get into the habit of winning the battle and selecting useless talents, it creates the potential for a vicious cycle of picks that won’t help the team.

Speaking of cycles, an owner and general manager often go through a couple of coaches before the owner decides to replace the front office head. It makes a prospect with long-term promise but short-term flaws more appealing in many a GM’s eyes and reinforces the problem I stated above.

Where do scouts fit into this equation? Based on what I’ve heard from former scouts with the Chiefs, Rams, Jets, Browns, Vikings, Cardinals, 49ers, Eagles, and Lions, the common theme is to speak when spoken to. And, if you’re going to speak, to pick your spots with care -- because you’re navigating a career minefield.

The reason is that most scouts are entry-level employees. According to ex-scouts, there is an unwritten truism called the 25/25 Rule, which describes the tendency for NFL organizations to fire veteran scouts and replace them with new scouts in their mid-twenties (25) at an annual salary of $25,000.

This practice often occurs when teams change leadership. It also keeps scout salaries low. However, the 25/25 Rule also creates an environment where scouts are more reticent to stand up for their takes on players. Working for the NFL is a dream job for many, and teams have made their job security too tenuous to engage in learning that could actually provide long-term value to the company.

The worst dynamic that can occur here is when a team’s general manager, coach, and owner have major philosophical splits when it comes to talent evaluation. This dynamic places scouts in the impossible situation of serving three masters. Or, worse yet, there is no defined system and it’s a free-for-all.

It’s the job of the general manager and owner to agree upon a vision for the product they want on the field. It’s their job to get a coach whose style fits with this vision. It all sounds like common sense, and it is. But it’s much harder to communicate and execute a vision –- especially when the owner and/or general manager has a tendency to micromanage in lieu of developing a sound set of processes.

One of the best examples where micromanagement dysfunction reared its ugly head and the media got a whiff was former scout Dave Razzano’s account of a disagreement he had with Rams general manager Charlie Armey over his scouting report of quarterback Alex Smith, as told to Yahoo! writer Michael Silver.

Razzano’s refusal to fall in line with the widespread belief that Smith was a big-time quarterback prospect led to a heated confrontation with Armey in a meeting at Rams headquarters a couple of weeks before the ’05 draft. Razzano’s report on the former Utah quarterback opined that Smith was "not as good as our backup, Jeff Smoker. Backup only for the Rams."

Armey, who declined to discuss the incident after it was initially reported by Santa Rosa Press Democrat’s Matt Maiocco, solicited the input of other scouts and coaches who'd studied far less tape (if any) of Smith, who ended up being picked No. 1 overall by the 49ers.

"There were 12 guys around the table, and Charley had them rate him on every attribute – arm strength; accuracy short; accuracy long; judgment; game management; ad-lib ability under pressure. And he put a highlight tape on the projector. I mean, obviously, he’s gonna be 30 out of 30, and every throw’s a great pass … it’s a highlight tape!

"He said, ‘Are you gonna sit there and be stubborn? Why can’t you see what we see?’ I got heated. I said, ‘I’ve watched seven tapes, and I’m not changing my grade.’ He told one of our assistants, ‘Go get all seven tapes.’ I started screaming, ‘You’re gonna look at highlight tapes? That’s how Akili Smith got drafted!’ [Scout] Tom Marino had me in a bear hug. I just lost my mind."

I would bet most teams don’t have communication issues this flawed, but it’s a great illustration of how the Rams process of talent evaluation was broken, mismanaged, or antiquated. I believe to varying degrees most teams are dealing with all three issues when it comes to training and managing scouts.

Develop a Trainable Process around the Vision

Based on the processes Parcells explained in ESPN’s 2011 NFL Draft Confidential and the conversations I have had with former scouts, I believe that, as knowledgeable as teams are about the game, their grading methodology is antiquated.

I also think it is a reason why NFL teams have ample room to reduce scouting mistakes, because this methodology creates a high level of variability from evaluation to evaluation. It also isn’t designed to help the team continuously improve its evaluation team:

Typical NFL Prospect Grading Scale

• 9.00 - A player for the ages (Jim Brown).
• 8.00-8.99 – A perennial All-Pro.
• 7.50 – 7.99 – Future All-Pro.
• 7.00-7.49 - Pro-Bowl-caliber potential.
• 6.50-6.99 – First-round-caliber player with Pro-Bowl potential.
• 6.00-6.49 - Potential to become a quality NFL starter.
• 5.50-5.99 - Potential starter and likely first-day pick.
• 5.10-5.49 – Potential to make an NFL roster and contribute.
• 5.01-5.09 – Has a better than 50/50 chance to make a roster.
• 5.00 – Has an even chance to make a roster.
• 4.75-4.99 – Training camp player.
• 4.50-4.74 – Potential invitee to an NFL training camp.
• 4.00-4.49 – Needs developmental time in another league.

On the surface this might seem like a clear scoring system, but it’s not. Ask a former scout with recent stints in the NFL and they explain that they watch the player perform, write some notes, and assign an overall grade according to these general definitions.

Now think about this point and the fact that most scouts are now young, inexperienced within the scope of their team, and likely to be out of a job within three-to-five years. Add to the mix that many scouts I talked to never had any training with the team. They often earn these jobs with an interview, a recommendation, and completion of a sample scouting report on a player.

The fact that this was enough to get started tells you that a lot of teams aren’t communicating their vision with enough detail to make a difference. One former scout for an AFC North team explained to me that he never had defined training for the position. New scouts were put to work and given the option to attend position meetings at the team complex when not engaged in 90-hour work weeks.

This is not formal training. Even if there is a 10- or 20-year veteran of scouting on staff, there aren’t a dozen of them who came up together, trained together, and have an exact working definition of the difference between a "6" and a "7" on this scale. The likelihood that they are going to work with everyone on staff and get the scouts on the same page is rare. Or, alternately, the methodology may be so flawed that it doesn’t take.

When individual evaluators have a different understanding of how a system is supposed to work because the system isn’t well-defined, differences of opinion will be rampant. None of these grade explanations on the scale above explicitly defines what NFL scouts should be considering when evaluating a player:

• Athletic skills (speed, flexibility, strength, agility, etc.)
• Position-specific techniques (pad level, routes, blocking, etc.)
• Conceptual knowledge of the game (vision, pocket presence, etc.)

The dynamic this absence of defined criteria creates within a team’s scouting department is similar to any business where there is a group of individuals of different ages, different levels of job experience, and different levels of knowledge about the job.

Think I’m wrong? Next time you’re at work, gather a group of your peers and ask them all to define a simple task in writing: How the receptionist should answer the telephone.

Then pretend you’re the receptionist answering the phone while they grade your performance using this 1-5 scale:

• 1 = Poor
• 2 = Fair
• 3 = Meets Expectations
• 4 = Good
• 5 = Excellent

Unless your company has very clear guidelines for this process, you’ll not only find that each of your peers has a different answer about how the receptionist should answer the phone, but they also will have a very different idea of how well/poorly the job was done. To compound the problem, ask them after the fact how they define each of these grades and you’ll likely get a different answer from each person.

This example underscores my point: a lack of clearly defined criteria increases variation among those assigned to judge performance.

Where one scout might define a player’s performance as "good," another may define it as "meets expectations." The difference between these two scores in the NFL might be the difference between a player projected to become a starter and one projected only to make a roster.

The best way to correct this is to fix the evaluation system. Here is how I would change the current grading system:

• Define what a team values from players: Make these definitions practical and accessible for every employee to apply when evaluating performance.
• Define the levels performance and tier the criteria: Use ranges from basic expectations all the way to what would qualify that player as a superstar in that skill set.
• Use a system of scoring that is weighted appropriately: The weight depends on the priority the team places on the quality they're assessing.
• Split the criteria into fatal and non-fatal errors: Fatal errors are flaws that your organization believes will either take too much time to correct or are too egregious to fix. One fatal error disqualifies that player from consideration. Non-fatal errors are fixable, but the team must agree on how many a player can have and of what kind to keep him in mind as a future employee.
• Score players based on meeting or not meeting expectations: First score players along the lines of fatal and non-fatal errors. Then, assess the level of ability among players who pass the primary evaluation.

I would hire a scout with enough experience and flexibility to lead the process of building this system and then have him train his staff on using it. This includes ongoing calibration sessions among scouts, position coaches, and general managers, where they watch specific types of plays prepared by support staff and discuss how they would define what they see.

If the outcome of the discussion leads to a change in the evaluation system, then they change it. The evaluation system should keep up with the innovation we see with strategy and personnel usage. This includes training to incorporate analytics that complement the evaluation process.

Will scouting become a science with these changes? Of course not, but the current grading system is low-hanging fruit as an opportunity to reduce harmful variations among those evaluating players. It also provides scouting directors and general managers a better understanding of the strengths and weaknesses of its evaluators and evaluation practices.

It may seem like a pain in the ass because coaches and general managers lack enough time as it is. However, in addition to what I propose below, the sacrifice of a public relations event or two a week could do the trick. And I’m sure most coaches would rather be doing something constructive to help their staff get better than glad-handing the public.

Keep Scouts at Home

Scouts have assigned territories. They drive from school to school during the season, attend practices, interview players, coaches, and trainers, and watch games. They do this not only for juniors and seniors, but underclassmen.

The amount of work is overwhelming. I also think it’s grossly inefficient. If the most valuable information comes from studying tape, sending a scout on the road is a waste of hours where they could be working instead of traveling.

A scout will see more with a focused session studying a player or set of players on a recording that he can pause, rewind, and fast forward. There will be times where scouts should travel to a school, but I’d make it the exception rather than the rule.

This also gives the scouts more opportunities to interact with the coaches, develop a better understanding of the team’s changing needs, and continually refine the criteria that the team uses to determine what it values in players at every position.

Hire More Scouts and Value Experience

I once worked for a manager who worked under Arthur Blank and Bernie Marcus before they sold Home Depot. He told me that one of the reasons this business was so good for a time was how this duo valued their front-line employees.

This regional manager told me that he had several front-line employees in his region that made two-to-three times as much as the average pay for the job. His staffs were knowledgeable, happy, productive, and efficient. The complacency concern was a myth he never saw become a reality until Blank and Marcus left and Robert Nardelli cut costs with the hope to raise profits.

Home Depot has been a much different company since, for both the employee and customer –- and not for the better. When the home improvement retailer finally ousted Nardelli, Marcus said its new leader "did not need fear and a hammer in his hands" to run Home Depot.

A wise NFL team would beef up its scouting department, institute training programs, and focus more on correcting mistakes rather than practicing a scorched-earth policy every time a new general manager or coach comes to town. If teams trained scouts and equipped them with the knowledge and tools to serve the changing needs of the team, they would save money in the long run when it comes to player investments as well as employee turnover (players, scouts, coaches, and management).

The goal isn’t for scouts to see every play of every player, but to make the most of every study session within the scope of what the team finds important. Good experience comes from correcting mistakes and giving employees the opportunity to apply those lessons. There’s no way many of these teams can develop good evaluation practices when it’s a revolving door of overworked front-line employees with little-to-no training.

If just a handful of teams invested in their scouting infrastructure to this extent, we might see a return of dynasties even in today’s era of free agency. If several teams did it, we might see a better brand of football. Or, at least, I could spend more time laughing when I read Mike Tanier’s April columns.

Matt Waldman authors the Rookie Scouting Portfolio, an online publication devoted to play-by-play analysis of offensive skill position prospects. Now in its eighth year of publication, the RSP provides pre- and post-draft rankings, analysis, and stylistic comparisons. Waldman also includes all of his play-by-play notes, grading reports, and a glossary that defines and explains his grading criteria. The RSP is available for download now and 10 percent of each sale is donated to Darkness to Light, a non-profit devoted to preventing sexual abuse in communities across the country.