It's been a while since I last posted. I'm posting this from my hotel room in Omaha at the AVCA convention, which is pretty cool.
Previous post: Serving Series Part 6: Generating Athlete-Level Data
Since I talked about how to collect data in the last post, today I’ll talk about what it can do for you, what it can’t do for you, and how to trust it. I’ll talk about how data can help you make better decisions, what a prescription from numbers cannot tell you, how confidence in your data can reassure you when the results of your decisions make you feel bad, and how buy-in from the people around you can protect your job.
When we’re trying to evaluate performance, cognitive bias can get in the way. Our brains aren’t very well suited to collect a bunch of data over time and then analyze it accurately. That’s what we made computers for. Not everybody’s cognitive biases are the same. I’m writing in this series to try to convince people who are very risk-averse from the service line to accept some more risk. But, there’s a significant contingent in volleyball who are too risk-acceptant from the service line. You might fall into one of these camps because of the biases you carry with you.
If you’re very risk averse, you might be overrating the badness of a missed serve because it feels bad to miss a serve so missed serves stick out in your mind. You might also underrate the badness of a perfect reception from your opponent because “at least we got to play” or by the time the point ends your focus has shifted from the server to how the block wasn’t closed or how the passer was out of position or how you just got beat by a good swing.
If you’re very risk acceptant, you might be carrying with you attitudes from the side-out scoring era that regarded a missed serve as almost no cost at all. As Chad Gordon says “Gotta think it would be tough to take down Karch on the big court in the 1993 AVP KOB in Daytona Beach, Florida if you’re just lobbing in freeballs from the service line.” You might also be too pessimistic about your team’s chances of winning a point when the other team gets to run their offense.
Everybody makes decisions based on the information that they have available. Cognitive biases make us look at that information in distorted ways. Two people can look at the same set of serves and outcomes and come to wildly different conclusions because of differences in how outcomes make them feel. Using carefully gathered data alongside the data that we’re always collecting with our eyes and brains can help us make decisions based on how likely we are to win the point and the game, rather than how we feel. When we use data to make decisions, we are trying to overcome cognitive biases by weighting events accurately in relationship to win probability. Analytics is simply additional information with which to make decisions.
Now that we’ve talked about what data-driven decisions can do, we have to talk about what they can’t. Data cannot tell you ahead of time what the outcome of a discrete event will be. If I say, “when we give up a perfect pass, we’ve got a 36% chance to win the point,” I am not saying “we can’t win the point if we give up a perfect pass.” I mean that, over the course of a bunch of points, we’ll win about 36% of the points in which we give up a perfect pass. Someone might respond “if you tell me that something is 36% to happen and it happens, will you admit that you were wrong?” I would say “no, things that have a 36% chance to happen happen all the time.” To which they might respond, “okay what would it take for you to admit that you’re wrong?” This is a more interesting question. If I have good reason to trust my data, I would never admit that I was wrong because I don’t lie. But, if I get significant new data that I have good reason to discredit my initial data or if I find out that the methodology I collected my initial data with was flawed, then I will admit that I was wrong because I was. When you’re introducing someone to data-driven decisions or learning about them yourself, avoiding these misconceptions is important.
Now I’ll talk about why trusting your data is important and how to trust your data. Because we are going against what we feel and what our brain is telling us is right, we’re going to end up making decisions that make us feel bad. Think back to the all-or-nothing serve example from this previous post. The numbers clearly say that serving aggressively in that situation is correct. But it’s a person better adjusted than me that wouldn’t think about that decision every day they’re sitting at home while their conference opponent plays in the tournament. Also, when you are the one bringing data to the team to make decisions, when those decisions don’t pan out, it’s natural for them to feel bad too. And, because it was your advice, it would be natural for them to feel betrayed by you. If we aren’t to be driven mad and driven apart by decisions not working out, we have to have confidence in our approach so that we can reassure ourselves that the decisions are correct, especially when they feel terrible.
This is why I talked about statistical significance in my last post. When you’ve made a decision based on data, and that decision doesn’t pan out, part of what you’ll end up dwelling on is questions like “was my data sound?” “did I have enough data to give me a good reason to make the decision I did?” “was flawed data why my decision didn’t pan out?” When these questions are swirling in your mind after losing a big game, you need to be able to look back at your data and look back at your decisions and at least know with the rational part of your brain that you made the right decision. When your data has significance, this is one metric that can reassure you that you’ve made a sound decision.
Finally, we have to talk about how to keep your job when you are the person giving people advice that has the potential to make them feel bad. Being the analytics person can make you a really hard person to be around, especially if you don’t lie and aren’t willing to let people you care about be wrong about stuff without saying something. I can’t really watch baseball anymore without getting frustrated at announcers that don’t understand analytics. I was watching the NCAA DI semifinals yesterday and getting frustrated at people around me complaining about the missed serves. If I had known them, I probably would have said something and disagreed with them and it probably would have been uncomfortable for all of us. If you’re a difficult person to be around, it can be hard to keep your job. When you have a different interpretive lens than your colleagues and you often give them advice that they see as blatantly wrong, it can be really hard to keep your job.
Often, we have to overcome cognitive bias not just to reassure ourselves when the result of a decision makes us feel bad, but also to justify our decisions to others. Analytically forward people are often attacked by traditionally minded people in sports. If your boss does not buy into the way that you make decisions, that can have ramifications for your job security. I am reminded of how people talk about data-driven decisions in football. When a team goes against the conventional wisdom and the play doesn’t go their way, coaches come under fire from the legacy pundits. When coaches make data-informed decisions that legacy pundits do not understand, legacy pundits will engage with those decisions based on scripts that are familiar. If the decision pays off, the legacy pundits say they “love the aggression” and “love that the coach trusted their athletes to perform,” during the game when it happens and don’t talk about it the next day. But, when the decision doesn’t pay off, the legacy pundits talk about how “the game is played by people, not computers,” and treat “analytics” as a four-letter word that doesn’t really mean anything but is meant to bring to mind a cabal of computer nerds dead-set on ruining football. Then, on the talk shows the next day, the chyrons say things like “ANALYTICS STRIKES AGAIN, DATA NERDS RUIN FOOTBALL GAME FOR EVERYONE.”
Luckily for most data-forward coaches in professional sports, the voices who do not know what analytics is or what it does are usually not the voices who affect their job security. Mostly these voices are coming from outside the building, not inside, so they can tune those voices out. But, if you’re trying to change a team or a department from an environment that doesn’t use data to a data-forward environment, you need to be prepared for how people who are not accustomed to responding to the results of data-driven decisions will respond to those results.
To be clear, analytically-backwards responses to data-driven decisions are not always malicious. In the football media, probably some of it is. But, when you’re talking to people in your department or on your team, negative and biased reactions to data-driven decisions are more likely to be informed by misunderstanding and people being made to feel bad because of things they don’t understand. So, you’ll need to be prepared to explain what data can and cannot do and dispel misunderstanding in a caring and delicate way.
You should also remember that being experienced in making data-driven decisions doesn’t make you better than people who are not. It doesn’t make you rational and smart and them emotional and dumb. It also doesn’t give you an excuse to treat others on your staff poorly. Recall the thought experiment from a previous post where an athlete is about to serve while the other team has match point and they can either serve with 10/10 aggression 50% ace, 50% error; or they can serve 1/10 aggression and give up a 2 pass every time. Let’s say I’m an assistant coach on that team. I haven’t talked about any of these numbers with my head coach and they believe in a more conservative serving approach. If I were to, behind the head coach’s back, tell the athlete to serve the 10/10, I would be wronging the head coach. Being right about the decision doesn’t excuse implementing it unkindly.
Comments