I have been putting this off but no more…
Data, Telemetry, Instrumentation, and Flighting are the new buzzwords…ok, at least the ones I am hearing about around the campfire at the moment. What is weird is that this has been part of what I do for the last 9 years or so. We have data coming out our ears at this point. What we have not had is an appropriate way to read and view what the data is telling us.
We create reams of data! What we suck at is is deciphering what the data is telling us and what to do with it! We have databases on top of databases collecting non PII (personally identifiable information) data that we could use to help us understand what our customers are doing and in some cases why they do it. For the life of us we cannot correlate that data into meaningful nuggets that tell us how to use it to make decisions in product development.
From the mountaintops we hear of these challenges to grasp and leverage customer data and telemetry but we need help! Like Security and Performance doing this job well is something refined over time with lots of failures, successes, and learning. You need to staff appropriately for these roles (statistics, DBA, ethnography, demographics, and cognitive learning). These take years and years of refining to get right and not just in one product development cycle. Just look at #Facebook as an example; how many times have they pissed off their user base because of a change they made based on telemetry and data? It’s one thing to look at your customer’s usage of your product and another to truly understand their intent and desire in daily usage.
Don’t get me wrong: I believe that data and telemetry used in product engineering and development separates the men/women from the boys/girls but it takes time and you need to do it right. You need to setup your databases to scale to the learning you want and to grow with the data you collect and then you need the flighting mechanisms necessary to do A/B testing appropriately. IT TAKES TIME! You can’t just jump into this game and you can’t just throw anyone at the problem; you need to be thoughtful, calculating, and deliberate. Otherwise you waste a ton of time and development effort on something that will never tell you what you want to know.
Do I have it all figured out? Absolutely not…but I do think you need to have an approach to this problem that is well thought through, intentional, and company wide. Learning a little at a time and delivering on the results you discover; however you need to be able to correlate across different parts of your ecosystem which we don’t do well at all today.
Flighting – this is hot! Also known as TIP or testing in production. Using concentric rings of exposure you try different ideas or even fixes to known problems to see how they fare before subjecting your whole general audience. This is brilliant and we need to invest in making this a normal way of doing business. We need to architect our software to adapt to this model and embrace it. Flighting combined with data and telemetry is powerful! Let’s do it!…but do it right.