We have been taking a step back lately and taking a little inventory. One of the things I have been suggesting is that as our ability to leverage data collected from usage grows the less dedicated testers we will need.
There are 2 primary objectives for a Quality Team, in my opinion:
- To validate and verify that what we have promised the customer, we actually deliver on.
- To exercise the product in such a way that we find the issues others don’t think about looking for (These come in the form of adhoc, perf, stress, longhaul, etc) – Other than designing Quality Assurance systems I think this is the most valuable deliverable of a test engineer.
The first bullet can be done almost entirely with data (telemetry and insights). We can add measures that let us know when the user (alpha, beta, customer or even automation) has executed a code path and whether we saw what we expected or not. This can be extremely useful for regression detection. There are some “gotcha’s” with this – you can inadvertently add to much telemetry making it difficult to find the nuggets because of all the noise. You can do this without a sufficient reporting system that doesn’t allow you to see the results of your data efforts.
The second bullet is where the true value of an Engineer with Test DNA brings their value. We just think differently and we will always find those crazy issues that show up only after the fact without good testing. Good flighting systems through rings of customer scope/scale help you find some of these problems through Beta testing but not always; it really depends how active your Beta community is. Really understanding how to put a product through its paces in adhoc (gorilla), perf, stress and long-haul takes experience and know-how.
So, my feeling is if you are able put the expectation on your development team for effective measures/telemetry, and you have a great pipeline and reporting system to effectively view the insights generated from your collected data, you can put more of the focus of a dedicated set of quality engineers on the second bullet above.
[REDACTED] – I actually had a whole paragraph written here with an example with something like ‘if you have roughly (t) testers supporting (d) developers you could draw down your test team by (n) percent if you introduced a great data generating, collecting, and reporting system’. Well, I am not going to break into jail by doing that but I do think you can significantly reduce your Quality Organization size with a very capable and effective data insights system implemented from automation all the way through to the shipped bits.
I’d love to hear some comments from those of you who have either seen this proven in action or debunked. Discuss – 🙂