Every 2 years or so, I get invited to participate in a “Software Quality” Initiative. Our company cares very deeply about the customer’s experience and we’re constantly trying to improve, so when someone has a conversation with a customer who has had a difficult problem, we try to find solutions that the whole company can improve from. Unfortunately, the approach is usually the same. People gather all the data that is currently collected and find a way to use it. The same people are pulled into the project every time and the same metrics are discussed…but little really changes.
There’s a lot of work in the industry and in academia to measure software quality but our team of experts doesn’t seem to take advantage of this progress. Instead of defining what quality is they focus on lists of bugs that they assume is what customers are concerned about. What’s worse is people consistently say that there is no way to measure software quality so they focus on what can be easily identified, which is bug information.
I was asked to join this initiative and thought, “How can I think about this problem differently?” I chatted with a statistician friend of mine who referred me to someone that specialized in software quality and he referred me to an industry expert. This person insisted that the software quality was a definable thing and that it can be measured. He recommended a couple of books on the subject and once I read them we had several conversations. It quickly became clear that software quality wasn’t as elusive as my colleagues claim, but it isn’t easy to define and measure.
There is one key element to software quality that most companies don’t want to come to grips with. The customers define quality, and that means that you have to define specifically who your customers are. Why this is important is because the first, and most important, aspect of the definition of quality is that your product meets the expectations of customers. So, it doesn’t matter what kind of bugs do or don’t exist if the product doesn’t do what the customer needs and/or expects.
After the customer’s and their respective expectations are defined, bugs then become important. What matters is the defect severity in relation to a customer’s business continuity and the probability that the defect will be encountered in production. In other words, what is the likelihood that a customer will be affected by the defect and how bad is it?
I would like to invite readers to share their experiences and approaches to software quality. This is a struggle for all companies and I would love to learn from your experiences. I haven’t seen any systemic approaches to measure product feature definition success. The measurements that I’m accustomed to hearing discussed are: Mean Time Between Failure (which is an adaptation of a hardware measurement that doesn’t seem to fit well), Mean Time to Resolution (for bugs), Time To Adoption (for customer’s acceptance of code into production), and Defects Per Lines of Code. While these metrics might be useful, they don’t really help me understand how well the customers can run their businesses with our products. Have you seen any approaches that have been effective in measuring Software Quality? Is there a better way to think about it?
If you don’t want your comments public but are willing to share your experience, don’t hesitate to email me at Email
I found a great blog article on the same subject: http://www.itbusinessedge.com/blogs/rob/?p=227
Customer’s always define quality.
If you talk to customers, the primary concern is the defect that caused a serious issue within their environment. But the bigger issue is always, what action was taken to ensure this type of issue doesn’t happen again. Was there a process improvement? If so where, in the requirement defintion phase, the development or testing phase, etc? Did the development methodology change to become more agile? Are we developing less between test periods? Did we implement a development methodology other than “code” and “fix”.
In a previous position, I spent a year researching how organizations quantify value to the business. The business being our commitment to customers and shareholders. Some people reading this may think shareholders is somewhat out of place. The though process is business is dimensional. The organization was focused on our external business which is our commitment to our customers and our internal business was focused on our commitment to our shareholders.
In my research efforts I started internally, looking for best practices. There was little information internally, so I sought out information from the industry. Talking to many forums, a number of customers, research market providers and their analysts.
I was emerged with information and the various development methodologies – “Code & Fix”, “Waterfall”, “RUP”, “Spiral”, “Hybrid methodologies” and “Agile (including most flavors)”. I went through the pro’s and con’s of each methodology, which companies employed what methodology, how they measured various points in their development process, etc. But regardless of the development methology, you are absolutely right there is no one single measure, let alone a development measure to help anyone understand how well the product is helping customers run their business.
Of the companies that emphasized value and customer satisfaction there were some commonalities in keeping a pulse on the components of their business. These organizations quantified value through a balanced scorecard. The scorecards all seem to reflect dimensional balance, such as -
-Internal measures and external measures
-Financial measures and non-financial measures
-Quantitative measures and qualitative measures
These organizations also had a scorecard at each level (top down) –
-Strategic
-Tactical
-Operational
These scorecards roll up from the bottom up. The measures were normalized to compare products and the measures were aligned with goals.
You know, I love all this talk of software quality but people really underestimate the amount of art that is still part of programming. While you can automate programming in much the same way you can automate writing, the creative elements remain elusive to capture and measure.
A truly gifted programmer can trash hundreds of lines of code and make a very compact, elegant, and speedy replacement for horrible code and guess what? She produces fewer kloc than her hack-neighbor. She also doesn’t resolve as many bugs since she collaborated with her test teams and hammered out problems before commit. In the metrics, she’s a slacker. Not much code, few bugs resolved. Disgusted, she heads off to marketing, where she can be appreciated for her knowledge of the product and ability to influence customers while lesser programmers with good metrics stay in the game (or get promoted to a place where they can hire more hack programmers).
I don’t know if there are good metrics to track and reward programmers versed in the art of programming, or what the best artist-to-hack ratio is for a specific programming team. I have, however, seen some of the best programmers I know leave the field because they can’t work in a metric-driven factory. Code is still produced, so is there a loss?
How do you retain the artist element of your programmer core? Do you need to?