The current NSCAA Soccer Journal published an abstract on the use of notational analysis by US soccer coaches. (Dr. M. Hamilton, “Current Practices and Perceptions of Notational Analysis among United States Soccer Coaches.”) The abstract provides data in table form collected from about a 1000 NSCAA members responding to a survey a few years ago. It provides no statistical analysis of the data or even information about the reliability and validity of the survey. Nonetheless the survey data is still interesting and inspired me to write about quality assurance and data collection. The paper uses the phrase notational analysis to mean “recording, analysis and use of team/player [match] statistics” and defined performance as “the game result (a win, draw or loss).” (p. 56). The paper then comments on the data using unsupported assumptions presented as reliable statements. (p. 56). In my opinion these comments are not merely unsupported, but erroneous. Despite the comments, the article is still very interesting. I prefer to call data collection, data collection, instead of some fancy title implying that the data collected is valid and reliable. Don’t confuse statistics, the raw data, with statistics, the mathematical analysis. There are two aspects to soccer coaching—training and managing. In both aspects there are various management tools that we can use to coach, the same tools available to manage a business. Most of you are probably aware that statistical analysis of player game performance data has been used to identify undervalued professional players (i.e., the book and movie “Moneyball.”) The same tools that an economist would use to identify undervalued stocks can be applied to coaching. If a coach, however, doesn’t know how to use statistical analysis to identify an undervalued stock, he won’t be able to use it to identify an undervalued player either. Quality assurance involves the management of processes through data collection and analysis. Management goals for commercial businesses typically are to reduce costs (in other words become more efficient) while maintaining or improving quality. The same can be said of coaching soccer. Coaches want to maximize the effectiveness of their training, reduce costs (in terms of time and resources), develop players and, of course, win matches. There are many, many tools available to managers looking to improve their processes and organizations. Over the years I have seen some business managers who have made effective use of some of those tools and others less successful. Some less successful managers have good intentions, but simply fail to manage the process well for various reasons. Too many other managers gather data and manipulate it to support a desired proposition—usually to “prove” that they are good managers and assign blame for any performance failures elsewhere. In either case the key is to understand the concepts of validity and reliability. For any analysis to be reliable, it must be based on valid and reliable data, valid in the sense that the data is accurate, reliable in the sense that the event is not an anomaly and would likely reoccur. If you have ever watched the instant replays and commentator’s cries after a controversial goal was allowed in a profession match, you understand the concept of valid data. There is an objective test for determining whether a goal is scored, but the referee’s decision is his opinion based on an objective test—not a fact itself. Referees are human and sometimes even the best make mistakes. Most players have experienced games when the ball seemed to be bouncing in your favor the entire game. Likewise most players have played games where you played fantastic but the ball always bounced in favor of the other team. From both those games we could obtain valid but unreliable data. Coaches shouldn’t change things based on valid but unreliable data. The paper makes unsupported assumptions disguised as true statements. It asserts that “notational analysis data” has greater reliability than “coach perceptions of game performance.” (p. 56). In making this error, the paper cites to a study that indicates that coaches can only recall critical game events afterwards 59% of the time and argues that coach perceptions of game performance are typically subjective opinions. Where is the citation to a study regarding the accuracy of “notational analysis data” to compare to the 59% accuracy rate for unaided memory? What relevance is the memory test or in-game data collection when matches are recorded by video tape? The author merely makes an unstated assumption that “notational analysis data” is a record of objective data and is accurate and valid. Elsewhere the paper notes that the majority of the time the “notational analysis data” is recorded with pen and paper by the coach or an assistant during a match. (pp. 60, 62). Recording an opinion on paper doesn’t make the opinion more valid. (Does the referee’s match report make erroneous goal decisions valid?) Subjective is not a synonym for inaccurate or invalid. As the point about the referee’s questionable decision made, objective opinions can also be wrong. While undoubtedly the validity and accuracy of opinions can vary, that is why you want trained and experienced coaches (and referees)! The paper also fails to account for the difference in qualitative analysis and quantitative analysis. Without specifying, the author’s definition of “notational analysis data” is quantitative information rather than qualitative information. Quantitative analysis is more useful for analysis of the tactical and physical aspects of a team’s match performance. Qualitative analysis is more useful for a technical analysis. For example while “notational analysis data” typical records the number of successful and unsuccessful passes, it doesn’t record any qualitative detail behind the conclusion that the pass was successful or not. If someone uses bad technique during a match, the important point is spotting the error and remembering what the error was, not remembering every occasion when the technique was employed. This leads to an important point. Match analysis, i.e., notational analysis, only becomes important with older players. For U-Littles where you are training the technical aspect, it is a waste of time. Whatever the result of the match, the coach should still be spending 4 years developing the same fundamental skills. A notepad and pen is all you need at a match to jot down notes on players and things to correct. While match results and player performance are unimportant to U-Littles, performance during training sessions is much more important. In training, players should spend 60 or more minutes on the ball compared to 5 minutes on the ball during a match. Match analysis is simply a distractor for U-Little coaches. In conclusion I expect that the article's comments on the data reflect conventional thinking in the upper levels of the US soccer community. One problem with taking an academic approach to a project is that an article must be conventional to be accepted for publication. Academic communities by their very nature are resistant to change and admission of error. Now for my views on the subject. If time and money were limitless, I would video tape every match and training session. I would use video tape to analyze and teach technique at every level. When I started teaching team tactics, I would use quantitative data and video to manage training and matches and teach tactics to players. When I started training the physical aspect if resources were available, I would eagerly use data collection to record heart rate during practices and games. The older the athlete, the more important data collection is to development of a physical training plan for that athlete. If I were managing a club at any level, I would use quality assurance tools in improve club processes and player development. I would not be merely collecting data without a specific purpose. What do you think about notational analysis? Afraid of collecting data because you don't understand how to do statistical analysis? I worked with accountants who performed statistical analysis by feeding samples of collected data into computer programs that handled the required mathmatics. The programer has to understand how to perform statistical analyses, not the accountant. So if you can't afford to hire an economist to analize your data, you could use a computer to handle the math.
Very nice write-up. I am a bit late to the party, but I'll share my thoughts. I coached Boys Varsity soccer for 8 years, took a "break" for 2 years at the Middle School level, and will be returning to Varsity this Fall. My school is a relatively small, private religious institution. If we have three club/travel players at once, it's a rarity, but we do have a solid level of athleticism and very committed and passionate players. I have my US Soccer "C" license for frame of reference and coached club for several years in addition to my school coaching. Towards the end of my first stint as Varsity coach, I began having a student manager track the following stats live during matches for each player: passes completed, failed passes, "other" turnovers, 50/50 ball won, successful tackle, and the standard fouls, offsides, goals, and assists. While I am under no illusion about the level of accuracy of the stats I got, having concrete numbers to tell players about passing % and turnover frequency was very powerful. Also, that little kid who worked hard but didn't do anything flashy and who previously got a ton of praise to other kids' confusion? No more confusion as I could now say, "Ian created 15 more possessions for us that half alone by winning loose balls and making tackles." Going forward, I intend to do something similar, but by using video through www.vidswap.com and the Pro iPad app from www.soccermeter.com. Neither of these tools will give me the specific numbers I tracked before, but Soccermeter provides outstanding team-based passing stats, and I can use video post match to identify key players' stats. Not having as many specific (and probably inaccurate) player stats right away may be a loss, but I believe it'll be outweighed by the benefit of real-time team passing info and the use of video to break down performances.
Yup. Probably not the right solution for club teams, but if you're a high school coach with the support of an avid booster club or a large athletic department, it's doable. Beyond the match analysis value it has great features for creating videos of clips from matches for highlights and breakdown videos that you can share with coaches and players pretty seamlessly.
Good discussion! Having had the opportunity to use ProZone, GPS/HR monitors, Vidswap and simple notational analysis I think that the recommendations of Hamilton are spot on. While you may feel Hamilton could have done a better job of citing sources or going into further detail with studies, the general consensus is that coaches (and humans in general) have very selective memory and are susceptible to confirmation bias. The use of these types of tools have been advocated and used by educators for years; and I feel should be used by coaches at all ages. Younger players can benefit from this type of analysis, especially if you involve them in the process. Some examples that I've done with younger players.... 4v4 tournament w/ 1 sub. Sub records shot location and outcome on a field diagram on a piece of paper. Empty circle for a shot, colored in circle for a goal. At the end the entire team looked at their diagrams and came to conclusions that there were better places to shoot from than not! Players were given a ticket for an end of the year raffle every time they did a feint during a game. Subs were used to help record them. This helped engage the subs in the game and also led to discussions on the bench as to when a feint was successful or not, and why. I think critiques like rca2's are important so that we don't completely rely on data for analysis as there are inherent weaknesses and biases in their production and collection, I feel this is a small price to pay. Don't throw the baby out with the bathwater! Tools like notational analysis should be in the toolbox of any modern coach.
I thought the second idea was very interesting. The first is teaching tactics rather than technique, but the second example is skill oriented. The same approach would work with any targeted technique (provided the player was only required to determine attempts rather than judge technique). Very good idea.
Tactics are just decisions... and it's important to start players early on with the decision making process. Many ways to give players the opportunity to think about and talk about the game, this is just another one I've used.
Sean, do you mind giving the lowest age and level of players that you use the tactical (first) example with?
The youngest I've done it with is U10. Any younger and you might run into issues with the spatial conceptualization, but if you felt your team might get into it then you could try it.
I found this link (on another Big Soccer forum) to an interesting white paper on a survey of the use of "analytics" at 25 professional clubs and leagues. The web site also has a ton of interesting information too. http://www.sas.com/en_us/insights/analytics.html http://www.sas.com/en_us/whitepapers.html http://www.sas.com/en_us/whitepapers.html#a See "Analytics in Sports: The New Science of Winning"