Please read the previous blogs to get some context to this post…although not entirely necessary it will help.
The last nine years at Microsoft (been here 14 but the last 9 have been in management) I have been working to achieve the ultimate dev/test engineering paradigm on the teams I have been Group Quality Manager (Bucking against the status quo). Working hard with my peer dev managers to push more of the build, component, and functional quality into the development team so that my Quality team can move to focus more on the customer scenarios and other difficult tasks such as Performance and Security.
I started on this endeavor in 2005 when I was asked to the be the Test Lead for an incubation team spun up to do a Microsoft Live Meeting lite which ended up being called SharedView. I led a very small team of Quality Engineers, half the size of the development team. We achieved our goals and slipped the service out on the web in a short 9 months where we then did quite a bit of development through TIP (Testing in Production). I then took a role on the newly formed Zune team where I again worked to build on the SharedView experience and do more to get closer to the dream.
Well, after 9 years of shipping product after product and iterating towards the ideal I believe I have come close. I currently have a team of 6 Quality Engineers supporting 18 developers. 80% of the component (unit) and functional tests are owned by the dev team. As the quality team we have taken a 2 pronged approach.
- Each Quality Engineer on the team has 1 or 2 consumers of our components as their ownership area. Those engineers are responsible for reaching out to those customers and understanding their requirements as well as expectations and then building the appropriate tests in our regression suite to verify those expectations and requirements moving forward. They have also documented this approach in a small Customer Quality Plan as a means of communicating and signoff with stakeholders.
- We (the Quality Team) look at the holistic System that is under our ownership and focus on high value, high impact, and fragile areas that have a high potential of disruption or failure under load or adverse conditions. We also take this opportunity to…wait for it…”do whatever it takes to get a high quality product out the door”. We are also working to identify the set of instrumentation that we can push into our telemetry infrastructure to learn as we start to roll builds out to end users. The levels of telemetry we focus on are Hygiene, Health, Insights Discovery, and Customer Understanding (more on this in another blog)
Several engineers on my team have taken on dev owned work items because at certain points in the product cycle we were short on dev resources and we had the ability and time to do it. There are other examples of how technical my overall team is and we are starting to break down this perception that I talk about in previous posts.
Microsoft is changing a lot internally when it comes to dev and quality roles/responsibilities. There is a big push for Quality to move to implementing telemetry and gathering data to help make product decisions. There is a large emphasis on Flighting – rolling out product changes to small customer groups and then adding larger customer groups as we gain confidence in the quality of the changes and receive feedback. It is quite possible that gone are the days of 8 week long test passes before shipping to the entire planet. Overall I think the move is good. I think the focus on data and telemetry is functional component specific; meaning that there may not be as much end user data collection at the hardware abstraction layer as there is in the Shell or UI. So painting quality with such a large data brush may not always be appropriate.
Everyday is a learning experience and I believe the best educational environment is one in which you feel underwater or under-qualified. Building something new and working to learn from your customer as early as possible is a big undertaking. We have to take what we know and turn it on its head; however that does not mean throw the baby out with the bathwater. Some of the moves that I have seen meant throwing away everything and starting over. I would rather we look at this opportunity of change as a way to make what we do more efficient and effective not an excuse to force a change for the sake of change.