Measuring hard-to-measure things

GitHub has an abundance of quantitative data about what people are doing. Over the past two years, it has built a practice of qualitative research dedicated to uncovering the why. Qualitative research surfaces blind spots with product and customers and has changed the way GitHub ships features. Using three examples, Chrissie Brodigan shares how GitHub rolls features out as controlled experiments.

Early product success is both exciting and terrifying because if the growth path is up and to the right, your audience will broaden, as will your challenges. Newcomers will arrive with experiences less similar to those of your earliest users and may begin using your product in ways you didn’t intend, leaving you puzzled. In GitHub’s case, there was a distinct shift in growth and audience: by 2013, new signups by developers with limited experience using version control and Git far outnumbered those by experienced developers. The cache of a GitHub membership within the developer community brought many people through the door and into a product that is frankly pretty hard to figure out. As a result, more people were failing than succeeding. All products have blind spots, and no amount of analytics and monitoring can account for hidden variables. The sheer volume of data we have now can make it hard for talented analysts to utilize information, and powerful tools and visualizations are rendered less useful. And even the best systems can gather data on unusual behaviors for a long time without signaling that something is amiss. At GitHub, with regard to engineering challenges, data collection, and analysis, we tend to think in terms of technical instruments. If you want to surface blind spots, you will have to get out from behind the numbers and in front of people. Think of yourself as a human instrument. GitHub’s Chrissie Brodigan covers three examples of how GitHub uses research to deliver product and connect with customers: how it measures new user uptake with longitudinal survey analysis and failure with exit surveys, how and why we roll out new features on GitHub.com (like pricing studies) as controlled experiments, and how it designs feature prioritization sneak attacks (learning through asking the right question) to learn more about customer culture.