You Too Can Be Masterful at Analyzing Data (Go Dirty)
“What is the most important thing you have learned about analyzing data over your career?” That was an interesting question posed to me while teaching a Master Class seminar on Design Research for Innovation to a group of senior researchers and designers last week in London. After some some reflection, I hit upon a simple idea.
The novice researcher is mostly concerned about collecting and organizing data well. We tend to rely on the presentation of the data, usually satisfied with finding some general observations that are well supported. After having considerable experience with this, we then start looking for subtle patterns in the data. As our career progresses we start to generate more nuanced insights by manipulating data in ever more sophisticated ways. And, importantly, we get better at deriving useful and relevant conclusions.
At some point we grow the confidence and skill to look beyond the “tidy patterns” (however useful they might be) and focus on the anomalies. We become fascinated by data that doesn’t fit the patterns, or that doesn’t support our hypothesis. What the beginner discards as noise in the data, the master focuses on. That is where the big “Ah Ha’s” are – and where the big proprietary insights come from that can drive innovation. It’s often in weird, dirty data that we make our best discoveries.
Questions to ask in the process might include: Why did just a few people do or say that? What is behind that? Is there a deeper, less obvious pattern that we can find if we really apply ourselves? The most exciting moments in research are when we uncover those tiny patterns that reveal useful complexities, like fractals that reveal the integrity of the whole.
The next time you sit down to analyze your data and you get it sorted out, put it away for a while. Go pour yourself another cup of coffee and come back to it. Focus on what doesn’t fit, what you have conveniently discarded. Your most juicy, significant insights are probably hiding there.