Writers of Pro Football Prospectus 2008

Most Recent FO Features

ManningPey98.jpg

» Game Preview: DEN-CIN

Maybe the Bengals are a paper tiger, but are they really that bad in prime time games? Is Peyton Manning struggling in Denver's new offense? We detail the Monday night clash.

17 Apr 2013

Futures: Scouting's Bum Rap

by Matt Waldman

Scouting gets a bum rap.

"Of course Waldman would say this," you proclaim. "He’s a scout!"

I may perform the fundamental role of one, but I am not a scout. This elicits laughter from my friend Ryan Riddle. The Bleacher Report columnist who holds Cal’s single season sack record and played with the Raiders, Ravens, and Jets says I have a misplaced sense of honor when it comes to refusing to wear that label.

I prefer talent evaluator, tape watcher, tapehound, or tapehead. My friends –- if I have any left since I started doing this work eight years ago –- might say "Film Hermit" is the best fit. I’ve never worked for an NFL team, so these names seem more suitable to me. Scouts have responsibilities that I don’t –- among them is reporting to management within a company structure.

If you have the chance to learn about the pre-draft process for most NFL teams, scouting is the study of a player’s positive and negative characteristics. It’s also an evaluation of how easy it is to fix the player’s issues and his potential fit within a team system. But based on what former scouts, coaches, and general managers of NFL teams say about the machinations that go into a team’s draft, I am thankful that I am not a scout.

While fans and writers may take the lazy route and blame picks gone wrong on poor scouting, it’s the general manager, coach, and owner who hold the weight of the decision-making power. This is a huge reason why scouting gets a bum rap.

To take it a step further, I’ll advance the popular Bill Parcells analogy of "buying the groceries." I can spend months in the grocery store and tell you that it has quality cuts of grass-fed steak; a delicious, rosemary batard baked in-house; and every variety of apple found in North America. But if those holding the wallet or cooking the food demand a papaya, I can tell them until I’m blue in the face that if they want a good one, it’s only found in Jamaica and they’re still going to pick an unripe one, take it home, prepare it, and then watch it spoil the meal.

It doesn’t help matters when I have to read Mike Tanier describe draft analysis as a pseudoscience. He’s right for the wrong reasons. Scouting is a craft, not a science. However, teams haven’t made it the same priority to address opportunities to improve scouting the way they have upgraded technology and embraced other forms of analysis.

With all the advances that the NFL has made with equipment, strategy, cap management, and technology, they haven’t done enough to advance the process of talent evaluation. It shouldn’t be the sports equivalent of Madam Zora’s, but until teams address the problems, Tanier gets to write entertaining draft pieces at their expense.

I think there is a lot that teams can do to improve their talent evaluation processes. What I will propose here are things I’ve learned from my experience in operations and process improvement. I base my solutions on problems I’ve gleaned in conversations with former scouts, reading and listening to former NFL general managers talk about their past roles, and extensive study of college prospects for the past eight years.

Some of these ideas may be new to the NFL, but I don’t begin to think they are revolutionary in the scope of other industries. I’m sharing these things because it’s too easy to listen to a gray-haired man in a suit on a television network and take what he says as gospel –- processes that may, in fact, be fundamentally flawed -- and then see that gospel perpetuated from generation to generation of football men.

When viewing NFL front offices and how they cope with change, I get the impression that many of them have a buttoned-up, low-risk culture similar in dynamic to Wall Street. It takes a lot for newer ideas to take hold in an NFL front office, as it does for an investment bank to accept "new blood" from a business school lacking a history of established connections with the firm as a personnel pipeline.

Some of what I’ll suggest is not even about new ideas; just better implementation of old concepts. The first point below is a good example where leaders tend to talk the talk better than they walk it.

Develop a Common Vision

Former NFL GM Ted Sundquist said on Twitter this week that there is a natural difference in preference when it comes to what personnel staff and coaches value in a prospect. Coaches gravitate to players with refined skills who are ready to play now. Personnel men and scouts are more forgiving of great athletes who can be "coached up."

The frame of reference each role has with its job security is a factor with these differences in views: coaches know they need to win now. The downside of acquiring a raw, athletic prospect is that this player may never learn at the pace to have a positive impact. If that prospect does figure it out, that coach knows it may not happen within a time frame that will help him keep his job.

Moreover, the new coach has now inherited a team with players that either don’t fit his system or haven’t demonstrated what it takes to produce on the field. Salary cap considerations become a factor in getting rid of that player.

Talk talent over need all you want, but what I’m really talking about is differentiating between usable talent and useless talent. If general personnel men get into the habit of winning the battle and selecting useless talents, it creates the potential for a vicious cycle of picks that won’t help the team.

Speaking of cycles, an owner and general manager often go through a couple of coaches before the owner decides to replace the front office head. It makes a prospect with long-term promise but short-term flaws more appealing in many a GM’s eyes and reinforces the problem I stated above.

Where do scouts fit into this equation? Based on what I’ve heard from former scouts with the Chiefs, Rams, Jets, Browns, Vikings, Cardinals, 49ers, Eagles, and Lions, the common theme is to speak when spoken to. And, if you’re going to speak, to pick your spots with care -- because you’re navigating a career minefield.

The reason is that most scouts are entry-level employees. According to ex-scouts, there is an unwritten truism called the 25/25 Rule, which describes the tendency for NFL organizations to fire veteran scouts and replace them with new scouts in their mid-twenties (25) at an annual salary of $25,000.

This practice often occurs when teams change leadership. It also keeps scout salaries low. However, the 25/25 Rule also creates an environment where scouts are more reticent to stand up for their takes on players. Working for the NFL is a dream job for many, and teams have made their job security too tenuous to engage in learning that could actually provide long-term value to the company.

The worst dynamic that can occur here is when a team’s general manager, coach, and owner have major philosophical splits when it comes to talent evaluation. This dynamic places scouts in the impossible situation of serving three masters. Or, worse yet, there is no defined system and it’s a free-for-all.

It’s the job of the general manager and owner to agree upon a vision for the product they want on the field. It’s their job to get a coach whose style fits with this vision. It all sounds like common sense, and it is. But it’s much harder to communicate and execute a vision –- especially when the owner and/or general manager has a tendency to micromanage in lieu of developing a sound set of processes.

One of the best examples where micromanagement dysfunction reared its ugly head and the media got a whiff was former scout Dave Razzano’s account of a disagreement he had with Rams general manager Charlie Armey over his scouting report of quarterback Alex Smith, as told to Yahoo! writer Michael Silver.

Razzano’s refusal to fall in line with the widespread belief that Smith was a big-time quarterback prospect led to a heated confrontation with Armey in a meeting at Rams headquarters a couple of weeks before the ’05 draft. Razzano’s report on the former Utah quarterback opined that Smith was "not as good as our backup, Jeff Smoker. Backup only for the Rams."

Armey, who declined to discuss the incident after it was initially reported by Santa Rosa Press Democrat’s Matt Maiocco, solicited the input of other scouts and coaches who'd studied far less tape (if any) of Smith, who ended up being picked No. 1 overall by the 49ers.

"There were 12 guys around the table, and Charley had them rate him on every attribute – arm strength; accuracy short; accuracy long; judgment; game management; ad-lib ability under pressure. And he put a highlight tape on the projector. I mean, obviously, he’s gonna be 30 out of 30, and every throw’s a great pass … it’s a highlight tape!

"He said, ‘Are you gonna sit there and be stubborn? Why can’t you see what we see?’ I got heated. I said, ‘I’ve watched seven tapes, and I’m not changing my grade.’ He told one of our assistants, ‘Go get all seven tapes.’ I started screaming, ‘You’re gonna look at highlight tapes? That’s how Akili Smith got drafted!’ [Scout] Tom Marino had me in a bear hug. I just lost my mind."

I would bet most teams don’t have communication issues this flawed, but it’s a great illustration of how the Rams process of talent evaluation was broken, mismanaged, or antiquated. I believe to varying degrees most teams are dealing with all three issues when it comes to training and managing scouts.

Develop a Trainable Process around the Vision

Based on the processes Parcells explained in ESPN’s 2011 NFL Draft Confidential and the conversations I have had with former scouts, I believe that, as knowledgeable as teams are about the game, their grading methodology is antiquated.

I also think it is a reason why NFL teams have ample room to reduce scouting mistakes, because this methodology creates a high level of variability from evaluation to evaluation. It also isn’t designed to help the team continuously improve its evaluation team:

Typical NFL Prospect Grading Scale

  • 9.00 - A player for the ages (Jim Brown).
  • 8.00-8.99 – A perennial All-Pro.
  • 7.50 – 7.99 – Future All-Pro.
  • 7.00-7.49 - Pro-Bowl-caliber potential.
  • 6.50-6.99 – First-round-caliber player with Pro-Bowl potential.
  • 6.00-6.49 - Potential to become a quality NFL starter.
  • 5.50-5.99 - Potential starter and likely first-day pick.
  • 5.10-5.49 – Potential to make an NFL roster and contribute.
  • 5.01-5.09 – Has a better than 50/50 chance to make a roster.
  • 5.00 – Has an even chance to make a roster.
  • 4.75-4.99 – Training camp player.
  • 4.50-4.74 – Potential invitee to an NFL training camp.
  • 4.00-4.49 – Needs developmental time in another league.

On the surface this might seem like a clear scoring system, but it’s not. Ask a former scout with recent stints in the NFL and they explain that they watch the player perform, write some notes, and assign an overall grade according to these general definitions.

Now think about this point and the fact that most scouts are now young, inexperienced within the scope of their team, and likely to be out of a job within three-to-five years. Add to the mix that many scouts I talked to never had any training with the team. They often earn these jobs with an interview, a recommendation, and completion of a sample scouting report on a player.

The fact that this was enough to get started tells you that a lot of teams aren’t communicating their vision with enough detail to make a difference. One former scout for an AFC North team explained to me that he never had defined training for the position. New scouts were put to work and given the option to attend position meetings at the team complex when not engaged in 90-hour work weeks.

This is not formal training. Even if there is a 10- or 20-year veteran of scouting on staff, there aren’t a dozen of them who came up together, trained together, and have an exact working definition of the difference between a "6" and a "7" on this scale. The likelihood that they are going to work with everyone on staff and get the scouts on the same page is rare. Or, alternately, the methodology may be so flawed that it doesn’t take.

When individual evaluators have a different understanding of how a system is supposed to work because the system isn’t well-defined, differences of opinion will be rampant. None of these grade explanations on the scale above explicitly defines what NFL scouts should be considering when evaluating a player:

  • Athletic skills (speed, flexibility, strength, agility, etc.)
  • Position-specific techniques (pad level, routes, blocking, etc.)
  • Conceptual knowledge of the game (vision, pocket presence, etc.)

The dynamic this absence of defined criteria creates within a team’s scouting department is similar to any business where there is a group of individuals of different ages, different levels of job experience, and different levels of knowledge about the job.

Think I’m wrong? Next time you’re at work, gather a group of your peers and ask them all to define a simple task in writing: How the receptionist should answer the telephone.

Then pretend you’re the receptionist answering the phone while they grade your performance using this 1-5 scale:

  • 1 = Poor
  • 2 = Fair
  • 3 = Meets Expectations
  • 4 = Good
  • 5 = Excellent

Unless your company has very clear guidelines for this process, you’ll not only find that each of your peers has a different answer about how the receptionist should answer the phone, but they also will have a very different idea of how well/poorly the job was done. To compound the problem, ask them after the fact how they define each of these grades and you’ll likely get a different answer from each person.

This example underscores my point: a lack of clearly defined criteria increases variation among those assigned to judge performance.

Where one scout might define a player’s performance as "good," another may define it as "meets expectations." The difference between these two scores in the NFL might be the difference between a player projected to become a starter and one projected only to make a roster.

The best way to correct this is to fix the evaluation system. Here is how I would change the current grading system:

  • Define what a team values from players: Make these definitions practical and accessible for every employee to apply when evaluating performance.
  • Define the levels performance and tier the criteria: Use ranges from basic expectations all the way to what would qualify that player as a superstar in that skill set.
  • Use a system of scoring that is weighted appropriately: The weight depends on the priority the team places on the quality they're assessing.
  • Split the criteria into fatal and non-fatal errors: Fatal errors are flaws that your organization believes will either take too much time to correct or are too egregious to fix. One fatal error disqualifies that player from consideration. Non-fatal errors are fixable, but the team must agree on how many a player can have and of what kind to keep him in mind as a future employee.
  • Score players based on meeting or not meeting expectations: First score players along the lines of fatal and non-fatal errors. Then, assess the level of ability among players who pass the primary evaluation.

I would hire a scout with enough experience and flexibility to lead the process of building this system and then have him train his staff on using it. This includes ongoing calibration sessions among scouts, position coaches, and general managers, where they watch specific types of plays prepared by support staff and discuss how they would define what they see.

If the outcome of the discussion leads to a change in the evaluation system, then they change it. The evaluation system should keep up with the innovation we see with strategy and personnel usage. This includes training to incorporate analytics that complement the evaluation process.

Will scouting become a science with these changes? Of course not, but the current grading system is low-hanging fruit as an opportunity to reduce harmful variations among those evaluating players. It also provides scouting directors and general managers a better understanding of the strengths and weaknesses of its evaluators and evaluation practices.

It may seem like a pain in the ass because coaches and general managers lack enough time as it is. However, in addition to what I propose below, the sacrifice of a public relations event or two a week could do the trick. And I’m sure most coaches would rather be doing something constructive to help their staff get better than glad-handing the public.

Keep Scouts at Home

Scouts have assigned territories. They drive from school to school during the season, attend practices, interview players, coaches, and trainers, and watch games. They do this not only for juniors and seniors, but underclassmen.

The amount of work is overwhelming. I also think it’s grossly inefficient. If the most valuable information comes from studying tape, sending a scout on the road is a waste of hours where they could be working instead of traveling.

A scout will see more with a focused session studying a player or set of players on a recording that he can pause, rewind, and fast forward. There will be times where scouts should travel to a school, but I’d make it the exception rather than the rule.

This also gives the scouts more opportunities to interact with the coaches, develop a better understanding of the team’s changing needs, and continually refine the criteria that the team uses to determine what it values in players at every position.

Hire More Scouts and Value Experience

I once worked for a manager who worked under Arthur Blank and Bernie Marcus before they sold Home Depot. He told me that one of the reasons this business was so good for a time was how this duo valued their front-line employees.

This regional manager told me that he had several front-line employees in his region that made two-to-three times as much as the average pay for the job. His staffs were knowledgeable, happy, productive, and efficient. The complacency concern was a myth he never saw become a reality until Blank and Marcus left and Robert Nardelli cut costs with the hope to raise profits.

Home Depot has been a much different company since, for both the employee and customer –- and not for the better. When the home improvement retailer finally ousted Nardelli, Marcus said its new leader "did not need fear and a hammer in his hands" to run Home Depot.

A wise NFL team would beef up its scouting department, institute training programs, and focus more on correcting mistakes rather than practicing a scorched-earth policy every time a new general manager or coach comes to town. If teams trained scouts and equipped them with the knowledge and tools to serve the changing needs of the team, they would save money in the long run when it comes to player investments as well as employee turnover (players, scouts, coaches, and management).

The goal isn’t for scouts to see every play of every player, but to make the most of every study session within the scope of what the team finds important. Good experience comes from correcting mistakes and giving employees the opportunity to apply those lessons. There’s no way many of these teams can develop good evaluation practices when it’s a revolving door of overworked front-line employees with little-to-no training.

If just a handful of teams invested in their scouting infrastructure to this extent, we might see a return of dynasties even in today’s era of free agency. If several teams did it, we might see a better brand of football. Or, at least, I could spend more time laughing when I read Mike Tanier’s April columns.

Matt Waldman authors the Rookie Scouting Portfolio, an online publication devoted to play-by-play analysis of offensive skill position prospects. Now in its eighth year of publication, the RSP provides pre- and post-draft rankings, analysis, and stylistic comparisons. Waldman also includes all of his play-by-play notes, grading reports, and a glossary that defines and explains his grading criteria. The RSP is available for download now and 10 percent of each sale is donated to Darkness to Light, a non-profit devoted to preventing sexual abuse in communities across the country.

Posted by: Matt Waldman on 17 Apr 2013

34 comments, Last at 16 Dec 2014, 2:43pm by ayunksyah

Comments

1
by robbbbbb (not verified) :: Wed, 04/17/2013 - 2:48pm

This is not only a good article on scouting, but a good article on business and employee relations. Some really smart team should make Mr. Waldman an offer he can't refuse to go run their scouting department and implement these principles.

2
by Anonymousse (not verified) :: Wed, 04/17/2013 - 3:19pm

I've always been amazed at how penny-rich/pound-foolish professional sports (and I guess big business in general) can be. Paying scouts (who are the foundation of your entire player acquisition system a below living-wage; MLB teams having their minor league players subsisting on pizza and chicken fingers when proper nutrition would probably improve all their performance; etc. I've seen companies put figures up like how it costs them $40K in recruiting and turnover costs to replace an employee, and then refuse to give employees cost-of-living raises. Its a bit crazy.

3
by trill :: Wed, 04/17/2013 - 3:25pm

Great article, Matt. I'd be really interested to see the differences in scouting/player acquisition process between teams that are perennial contenders and those that are not.

5
by Bobman :: Wed, 04/17/2013 - 4:04pm

As I read it, the Nardelli critique slammed the experience for employees and shoppers, not shareholders. Those two things can move in opposite directions (short term). Ultimately the people the CEO/chairman (every employee, arguably) is responsible to is the shareholder/owner. Now the owners (and the CEO who is their rep), if they value a sustainable cash flow and reputation, would be wise to say good customer service has value too, even if today's profits are lower, it tends to contribute to more long-term customer relationships.... If they view the company as one that exists purely to build sales and profits by saving their customers a penny on a length of lumber, then the only thing that matters is low costs. If they expand their model and think they're selling expertise and bulding loyalty, etc, then costs aren't everything.

11
by Dan Slotman :: Wed, 04/17/2013 - 6:25pm

This view of publicly held companies is common, but particularly toxic. The company is to be run for the benefit of shareholders, true. But your argument stems from an unstated assumption that assumes that a shareholder now won't be a shareholder in the future. Running a company for short-term profit hurts long-term shareholders.

However, corporate governance is structured much like the NFL. There is a lot of turnover, immediate results are valued very highly, and memories don't run long. Arguably it is worse than the NFL because a lot of people sit on boards of directors at many firms, leading to obvious conflicts of interest and opportunities for collusion.

17
by Rich A (not verified) :: Thu, 04/18/2013 - 12:29am

Interestingly enough Britain has rules regarding stock ownership in corporations that only allow stock owners to buy stocks tied to a time frame. I don't think this is a blanket over all of business but only certain lines of industry. Further, I'm pretty sure that some corporations have by-laws that state similar things even if they are publicly traded so that control of their boards is always tied to long term health of the company, something along the lines of "anyone who owns more than 2% in a company is required to own those shares for a minimum of 5 years." That's not an exact quote but more along the idea of how they operate.

In my casual reading of business magazines I've never seen the same kinds of things discussed about American Business. It is almost assumed I think that in America stock trading and business ownership is more of a job that's a high stakes game than an actual ownership of a company. I could be wrong though as I'm Canadian and what do I as a Canadian really know about corporate American finance.

20
by Mr Skinner :: Thu, 04/18/2013 - 4:50am

Afraid that is not true- UK short termism re investment is as bad as the US. There are no laws requiring you to hold a stock for any length of time (unless you are an employee).

27
by IrishBarrister :: Thu, 04/18/2013 - 1:57pm

I take a break from work to check FO and, hey look, securities and corporation law (I'm a business attorney). Without going into too much detail, statements like "There are now laws requiring you to hold a stock for any length of time" strike me as the football equivalent of someone saying "Defensive backs can't bump a receiver." Well, there's bright line rule within the first five yards, and while technically true after that, it gets kind of gray by what you mean 'bump', and then it gets even messier when you're going for the ball ... I'm not trying to be insulting or anything, but if securities law was that simple, Goldman Sachs and the SEC wouldn't need teams of attorneys to hack these issues out.

6
by Bobman :: Wed, 04/17/2013 - 4:09pm

Rob, I loved this article and felt it had a ton of insight that most of us did not have before. the 25/25 rule? i can totally see it but it's crazy. I assumed the scouting depts probably had one grizzled 20-year vet, a couple 5-year vets, and a swarm of sharp kids feeding something coherent and comprehensive to the GM who pulled the trigger. Looks like it's a little less structured than that, and teams suffer unless they have a couple bright stars in the mix.

Reminds me of the Bucs' clever way around the salary cap when they hired Gruden for $7 Mil. The logic being (fairly novel at the time) you are lmited in what you can spend on your team, so to get a primo competitive advantage (and if you are willing to spend) buy the best staff you can afford. Sadly, that might stop at the top 1-2-3 coaches on the roster, but should probably filter all the way down to the scouts, trainers, equipment guys, etc for long-term organizational health. Maybe fewer dollars of profit, but more wins over the course of a decade...?

7
by Kaptain Kirk (not verified) :: Wed, 04/17/2013 - 4:09pm

Very nice Matt. I liked Mike Tanier's article too. I read it a few days ago. Everything I've studied about the draft, combine, scouting and Front Offices is summed up quite nicely in your article.
Thanks.

8
by JonFrum :: Wed, 04/17/2013 - 5:11pm

This article assumes that all scouts do is assign number grades. Belichick is described as requiring each scouting recommendation to say who on the team's roster the new player would replace. So Patriots scouts need to know the team roster, and be able to project college players on to it.

As to the idea that all you need to know is on the tape - really? I want to know how much time a prospect spends in the weight room, and how he interacts with coaches on the practice field. The world is full of talented bums - you need to weed them out.

And just sayin' - it would never occur to me to call a blogger a scout. A scout could blog his work, but a guy selling a draft report is not a scout.

9
by Whatev :: Wed, 04/17/2013 - 6:15pm

Except, the Patriots are the exception, not the rule. And I don't think it's being argued that all you need to know is on the tape, only that a lot of what you need to know is on the tape. Traveling should be the last step in evaluation because it's the most costly step, both in terms of time and money.

13
by Matt Waldman :: Wed, 04/17/2013 - 8:45pm

I agree, but I see writers refer to themselves as scouts and other writers refer to bloggers like me as scouts. It's inaccurate and one of the more outspoken about saying so.

I'm glad you would never think of it. I wish more people were like you in that regard.

Thanks

22
by Anonymousse (not verified) :: Thu, 04/18/2013 - 9:09am

"As to the idea that all you need to know is on the tape - really? I want to know how much time a prospect spends in the weight room, and how he interacts with coaches on the practice field. "

But you're not going to get that traveling to the player's location for a weekend. Players/coaches/etc behave very differently when scouts are around.

26
by Aaron Brooks Go... :: Thu, 04/18/2013 - 1:04pm

Head coaches do.

Position coaches, though? Not as much.

10
by theslothook :: Wed, 04/17/2013 - 6:15pm

One interesting comment I want to make on this topic. If we think about scouting and how it improves over time, one might suspect that the inherent draft variance should be lower in current drafts relative to prior one's. I decided to test for this using a simple structural test.

Interestingly, it turns out that the difference in variance across drafts is not significant enough; however, the significance level has been markedly decreasing with time, implying that maybe scouting has really really slowly been getting better, but not enough.

12
by Karl Cuba :: Wed, 04/17/2013 - 8:33pm

I think you might be nearly saying something interesting but you haven't quite spelled it out. Variance of what across drafts?

14
by Matt Waldman :: Wed, 04/17/2013 - 8:51pm

What I'm discussing is variance of knowledge, which then leads to variance with how to score a particular skill set. You want your front-line employees, middle management, and upper management to have as close to the same knowledge of what the team values and how much it values aspects of performance so when they evaluate a player, the spread in takes isn't that one sees a starter, the other sees a backup, and the third evaluator sees a UDFA and they all supply different reasons that the other hadn't factored into the evaluation.

This instance should be the exception and then used to improve the process of evaluating that position or situation in the future.

15
by theslothook :: Wed, 04/17/2013 - 9:03pm

@Karl - The variance I'm referring to is variance of draft performance. A while back for school, I created a draft value index similar to the one danny's been running. Basically, if scouting is more accurate now, then the variance should be decreasing across time at a significant level. It hasn't been, but that doesn't mean it hasn't been decreasing

25
by Karl Cuba :: Thu, 04/18/2013 - 11:03am

So are you looking at variance of talent in a particular drafts or the variance of the distribution of talent within particular drafts? I would expect the talent levels of drafts to vary year to year anyway.

16
by theslothook :: Wed, 04/17/2013 - 9:04pm

@Matt - curious, if you examine how the draft variance breaks down across picks. It is always the case that the earliest picks have the highest degree of variance and then it decays slowly as you go further down the draft. This feels a bit counter intuitive, especially since the early picks are the most heavily scouted.

18
by Rich A (not verified) :: Thu, 04/18/2013 - 1:44am

Really interesting read Matt, thanks for posting it.

I was recently having a discussion with a buddy of mine who is doing a professional accounting designation at the major firm PwC (PricewaterhouseCoopers Professional Services (Auditing, Taxes, Counsulting)) and the fact that he works from 8:30 in the morning till 10:00 at night and that if you aren't prepared to work that kind of time that you will be immediately let go and you will never get your designation. This system is an intake of new recruits who are then given a bunch of grunt work under a team leader who is a partner and a senior associate who is coach to a few of the recruits. Often those people who are successful in attaining their designations are recruited to the companies they are consulting for to work as senior strategists and are placed on the boards (My sister got her designation at PwC and then left a year later for a large corporation where she works 35 hours a week for nearly 6 figures, which is about half of what she'd make at PwC if she was still there but she'd have no life then). After hearing about his work I came to the same conclusion as Matt; isn't it grossly inefficient to always have to be training new recruits. Though in his football example, it doesn't sound like there is much training going on.

I've even read War Room by Holley and so I know about how the Patriots have some systems in place but really it seems like there are more structures of ideas with the Patriots rather than actual systems of objective measurements. That being said, I'm sure they have a system, it just isn't revealed, nothing about the flow of information in that team is not controlled.

Getting back to the professional firms, I'm entering law school and so I know that the professional firms are actually mimicking a development from law firms. Looking into this style of coaching up new recruits before anyone else can have them and training them up specifically in the companies own system was started about 100 years ago by the Cravath law firm in New York. Reading into the theory about why this system was developed was really interesting. Cravath, a partner at the firm, wanted to hire the best talent - not the hardest workers, develop them to be extremely productive and efficient in the Cravath environment and protocols, which were themselves continually evaluated and revised, and then promote them into leadership positions in the firm. If a entry level associate wasn't improving they would be chucked. If a person wasn't growing: continuing to evolve and grow after being a senior associate, they would also be chucked. Basically, you had to be such an independent thinker that you were continually bringing valuable ideas to the firm. That value wasn't just production but creation.

Which brings in why Google is such an interesting company. Each Friday is dedicated to pet projects there that Google owns but each person can run themselves or chooses to participate on. It's not handed down from above but is a grassroots creation. This team system is actually completely how Gore (makers of Goretex) operates, new uses for their technology are constantly being brought up by anyone that has the idea.

Of course though what I described in the first paragraph isn't exactly what I described when describing Google but that's where the modern Cravath system is different from the original vision. Cravath actually placed a premium on what Valve software calls T shaped individuals. Individuals who have broad interests and talents as well as one specialized talent. Cravath thought that if a person was spending all their time working they would stunt their own growth and that lack of growth at the individual level would hurt the corporation. In the original model the firm was the best firm because by hiring the best talent and doing the work of weeding out the poor workers and also training them led to highly developed workers who were happy and profitable at the firm which was good for the workers, the firm, and the clients. Often those who were chucked out in the 'out or up' policy were able to get onto strategic teams at other corporations as even being hired (and fired) by the Cravath firm led to that kind of prestige.

This leads me to one key question of Matt and his theory regarding a new shape for scouting. Matt posits that individuals are not able to speak freely like at Google or Gore for fear of losing their jobs but he is also advocating a Cravath (modern) system that is more rigid with how scouts report. I can see how there should be more structure than the simple number structure with adjoining numbers but by starting down a road of structuralism won't the scouts be further hemmed into a fear of speaking their minds? Won't the rules for scouting and the system eventually be a weight because the system starts becoming dysfunctional itself because it can't capture what it needs to?

It can't capture the intangibles. But that's why their intangibles. It's one thing for the combine to measure numbers and for that to be spit out but how do you measure things like hunger for hitting? passion for the game? Gamespeed vs trackspeed? Those things have to be described in words or simple checkmarks (pretty close to what Matt said actually regarding fatal criteria). But even then, if they're checkmarks, will it be nuanced enough, it's one thing for a linebacker to like hitting wide receivers and it may be another thing to take on the fullback in the hole.

I think this brings up an interesting discussion regarding EA sports and the simulation of football based on their 30 or so scouted variables of the 1500 or so players in the NFL. EA and Tiburon know how their simulation works and so when they say Marshawn Lynch has just as high of a break tackle as Peterson than I have to believe them for online play but for franchise play do I go in and change player ratings? Do I make Watt and Gronk the beasts that they truly are on the field or play according to EA's scouting? Do I trust my own perspective or do I trust someone who knows what the system is looking for from each position and say Gronk is as strong as a linebacker and as fast as a safety, he's not as slow as a linebacker and as strong as a safety. Does EA have a better scouting department than some NFL teams?

But I think the most interesting thing was something that the games had in the mid 2000's, player roles, and specifically Captain Comeback. If a QB was losing in the 4th and had this role his stats would jump. Except, the only way to earn this role was to play so bad in the first 3 quarters that his team would be losing going into the 4th. But it's a simulation, so how does a QB play bad and then win games in the 4th quarter. Does the game have some randomizer that say's this QB is named Tebow and so he will just be a winner? I don't think so, it's based on several calculations of QB awareness vs DB awareness modified by coaching attributes. It's like EA recognized that they could have 25 variables for a player but that even with 25 variables they wouldn't be able to fully describe how a player fits onto the field and what kind of role he has, they had to come up with these 20 or so specialized roles that more fully fleshed out situational play. EA had a system for scouting but they wanted it more descriptive. I think they wanted to balance the fact that players are measurable in certain respects but that individual uniqueness cannot always be described within such a system.

Sorry that that was all over the place. Basically, how does a system capture the true uniqueness of a player without resorting to words which are somewhat flawed? Welker who was so small and shifty, Bruce Irvin who was so quick to accelerate, Watt and his tenacity. All these words convey ideas to you but your idea of and my idea of tenacity may be understood differently. Until there is a proto-language and telepathy I think there will always be this inability to communicate perfectly. Though I think there can certainly be strides made with scouting, it certainly seems like whatever Baltimore does is close, and same with New England, as long as it isn't on the perimeter.

19
by JDW (not verified) :: Thu, 04/18/2013 - 2:26am

What?

28
by Matt Waldman :: Thu, 04/18/2013 - 2:28pm

This leads me to one key question of Matt and his theory regarding a new shape for scouting. Matt posits that individuals are not able to speak freely like at Google or Gore for fear of losing their jobs but he is also advocating a Cravath (modern) system that is more rigid with how scouts report. I can see how there should be more structure than the simple number structure with adjoining numbers but by starting down a road of structuralism won't the scouts be further hemmed into a fear of speaking their minds? Won't the rules for scouting and the system eventually be a weight because the system starts becoming dysfunctional itself because it can't capture what it needs to?

It can't capture the intangibles. But that's why their intangibles. It's one thing for the combine to measure numbers and for that to be spit out but how do you measure things like hunger for hitting? passion for the game? Gamespeed vs trackspeed? Those things have to be described in words or simple checkmarks (pretty close to what Matt said actually regarding fatal criteria). But even then, if they're checkmarks, will it be nuanced enough, it's one thing for a linebacker to like hitting wide receivers and it may be another thing to take on the fullback in the hole.

Here's where I think I can clarify some things for you. The system I'm advocating actually requires interaction, discussion, and change to make it work. This is where I believe your assumption jumped off to another conclusion. The system is designed to be amended through continual watching of tape as a team of evaluators. Whether you do small bits daily, meet about it weekly, or have a longer session monthly, the process would be designed to spot where the structure needs to be updated. The goal is to maintain a living, breathing structure.

If you define the criteria then you can go a lot farther towards measuring things like comfort with physical play or other "intangibles." The time spent discussing how to define factors is critical when creating and maintaining a system. Where teams in any environment fail is when they treat a tool like the way many patients treat a pill prescribed from the doctor. They don't follow the process of changing diet and exercising. They just pop the pill and often enough not much changes with the original condition.

This is true of any NFL team that would view this as "changing from a number system to a yes/no system" but not embracing that this is just one tool in a bigger process where the benefits from the tool come from building it and maintaining it as a team.

What I mentioned in the article are just a few things - low-hanging fruit - there are other aspects to the evaluation process that I would implement not mentioned, but I wasn't writing the A-Z guide here, so it's understandable a question like this would arise.

But bottom line? You'll never have perfect symmetry across in-house evaluators in terms of knowledge and communication, but you can get a lot closer than they are right now.

32
by Rich A (not verified) :: Thu, 04/18/2013 - 4:44pm

Thanks for the insightful reply that fleshed out some questions I had regarding the system being a handcuff of sorts.

21
by SackSEER :: Thu, 04/18/2013 - 8:47am

This is a truly excellent article with a lot of insight. Kudos.

It made me revise some of my assumptions about the scouting process. I always assumed that the NFL Draft produced its share of busts because the inquiry itself was so difficult, not because of widespread industry dysfunction. In retrospect, I have been routinely disappointed by hardcore personnel guys that then become talking heads. I am often hopeful that they will provide a smarter perspective on the process, but they end up making comments that leave me scratching my head.

I have thought for a few years that scouts should be assigned by position rather than region. For one, the differences between a tackle and a quarterback are as different as a hockey defensman and a power forward in basketball. If scouts specialized by position, they could more quickly gain an expertise. Also, if I am the GM, and I want to know how the quarterbacks rank, it seems that a list if reports from the same person would be more useful than a mish mash of reports from different individuals, which would introduce a lot of noise that would have more to do with who graded what guy than the prospects themselves.

Also, I had seen that numbering scale before and thought it was ridiculous, and i never imagined it was something people in the NFL took seriously.

-----------
Sorry JPP!

24
by Karl Cuba :: Thu, 04/18/2013 - 11:00am

Your proposal for position focused scouting runs counter to the NFL's system if having scouts work through geographical regions and is another reason why scouts should be kept closer to home as Waldman suggests.

You could even go a step further and keep your scouts at home and send private investigators out to check prospects backgrounds while sports psychologists interviewed the athletes to establish work ethics and other intangibles.

30
by Matt Waldman :: Thu, 04/18/2013 - 2:31pm

Bingo. Scouts shouldn't have to be both evaluators of 22 positions and PIs and job interviewers. Let them do one thing and get a lot better as a team doing that one thing.

29
by Matt Waldman :: Thu, 04/18/2013 - 2:30pm

Thanks and I think those points about position assignment are interesting and have some validity.

33
by Scott C :: Thu, 04/18/2013 - 4:44pm

You would want more than one person's opinion on say, QBs. But you would want some specialization. If a scout has a talent for spotting strengths and flaws in how defensive players are in coverage, perhaps they should focus on that across all defenders.

Finding the right roles for people tends to result in higher overall performance than finding the right people for roles.

An assembly line would not work well if everyone was trained in every part of the line and was randomly assigned to a role each day. It also does not work well if you have no redundancy or adaptability in role assignment.

23
by cisforcookie (not verified) :: Thu, 04/18/2013 - 10:28am

there's an easy explanation for why things don't, and won't change. the people in charge of nfl front offices are incompetent, insecure clowns with major consistency bias issues. Someone in that position is unlikely to be open to radical thinking, even if they were capable of it. and since every organization is basically run this same way, there is no pressure to force out incumbents. And since incumbents select their own lieutenants, the pool of future incumbents is likely to have the same flaws.

31
by Matt Waldman :: Thu, 04/18/2013 - 2:32pm

Blunt, but I agree there are probably people exactly like this because they exist in every other industry.

34
by cheap ray bans uk (not verified) :: Wed, 01/15/2014 - 5:45am

Parlar | El cam