Software Metrics (cat flame > /dev/null)

When, in the past, I've been a software lead, the only two metrics that I found of any fundamental value were "it makes Tim happy" and "it makes the boss happy". I'll admit when I have a coworker who's way better at software than I am, I'll surreptitiously use "makes Nancy (or Bill)* happy" to heavily weigh "makes Tim happy".

I think the real problem is that you're trying to make a metric that somehow maps to "makes the project complete on time and under budget" or "makes the company money" or "saves the company the embarrassment of appearing on the 6:00 news hour next to a picture of a smoking hole in the ground". But those metrics are really difficult to map, so people end up using metrics that are -- however well intentioned -- smoke and mirrors.

• These are actual names, and if one of you is reading this you know who you are, and thanks for making me a better software engineer!

--

Tim Wescott
Wescott Design Services
http://www.wescottdesign.com

Do you need to implement control loops in software?
"Applied Control Theory for Embedded Systems" was written for you.
See details at http://www.wescottdesign.com/actfes/actfes.html

This shows both sides of the issue, IMO.

On the one hand, the "numbers" one comes up with are often only relevant in that particular Industry and/or organization. I.e., even if you settle on a *particular* ("standardized") metric, comparing observations of that metric in a desktop application vs. an embedded vs. a real-time, etc. application is essentially meaningless. While the metric might be "standardized", the problem to which it is *applied* renders it apples-and-oranges.

On the other hand, without *some* numerical "score", there is no way for an organization to evaluate its own progress. How do you know if quality is improving? Or productivity? etc (depending on what your actual goal happens to be).

It's hard to draw a parallel to any other aspect of a business. E.g., imagine if *your* accounting department tracked everything in terms of dollars... and another accounting department tracked everything in terms of loopholos. I.e., comparing between departments is meaningless (since loopholos and dollars are orthogonal measurement units) -- yet, comparing current to previous within a department *does* have value!

IMO, the actual metric(s) chosen have to be *easy* to measure unambiguously (automated), not easily "subverted" *and* only used for relative comparisons within an organization/individual (i.e., a person writing device drivers would exhibit different metrics than a person working on GUI's -- even within the same Industry/Organization)

Op Mon, 11 Jul 2011 23:30:43 +0200 schreef Don Y :

Do statistics have a purpose? Does telemetry have an inherent purpose?

Clarity, maintainability and testability, usually.

If the metrics are bad, then the code is likely to be bad. Bad code is a liability.

Use it to discipline developers. Either by showing the bad numbers or if they don't listen, by whacking them with the 3-ring notebook.

I don't understand that. How can a measurement of one metrics group affect the results of other metrics groups?

Metrics provide objective insight, that can be debated and reasoned, unlike subjective insight. It allows prioritization of coding and testing effort.

But in the end, all decisions are subjective. They come down to an incomplete view of the product development effort that will make some random human feel good enough about a certain decision.

--
Gemaakt met Opera's revolutionaire e-mailprogramma:  
http://www.opera.com/mail/
(Remove the obvious prefix to reply.)

Sorry, I meant "folks *pushing* for the metrics don't have a clear understanding of what they want to get *from* those metrics. I.e.,

*what* specifically are you trying to measure and what do you want to do with the data?

Or, productivity, complexity, etc.

But what constitutes a bad metric? Too few LoC/man-day? Too many function points? etc.

This dovetails with the above question...

Ha! I actually don't think they know *what* they want the numbers for (which is why they don't know *which* numbers they want!) -- but, somehow, think having numbers is better than *not* having numbers...

If you are trying to measure productivity, people will learn to "game" the numbers to keep their totals "up": LoC/man-day. Does this come at the expense of *quality*? I.e., is a buggy-LoC the same as a nonbuggy-LoC? How do you differentiate?

Is the developer keeping his productivity numbers up but at the expense of long-term maintenance issues? etc.

Yes -- assuming you have a stated goal/objective.

That reflects the bean-counter aspect of metrics and why they are so often disparaged. I believe there is a place for (various) metrics in the development process. But, doing so without adequate forethought and planning (and buy-in) makes an arbitrary imposition just a thorn to be worked-around.

I suspect everyone moaned about being "graded" on papers, exams, etc. in school -- but, no one (realistically) would claim those grades were without merit (?)

I don't think that you'll find many people suggesting that LoC/day or even function points is a useful metric, these days.

I think that the shape of the number-of-open-issues vs time graph is proably a fairly reasonable handle on project trajectory. Often goes something like a bell-curve. When issues are being closed faster than they're being opened you're probably more than half way there, and as it tails asymptotically towards zero you're nearly there...

Of course you need to be (diligently) using an issue tracking system to be able to easily use that metric.

A well maintained regression/unit test suite run automatically on check- in or overnight should keep a lid on the "buggy" issue.

Cheers,

--
Andrew

Indeed, they are just numbers and not a goal in itself. As usual.

Negative LoC change can be good if it reduces dead or duplicate code. Positive LoC change while not changing functionality can be good if it increases clarity. Metrics should always be combined to produce a useful indicator into an aspect of code quality, like clarity, maintainability and testability, or into productivity.

The developers themselves may not have any interest in the numbers, but the higher-ups certainly do.

As Andrew said, LoC/man-day is just one indicator of productivity. As you are hinting to, more factors should be weighed in.

If maintainability is not less important than productivity, then why is the developer focussing so much on productivity?

Not necessarily. Will the testers start testing any code, or will they start with the highest 'testability' number and give the developers some time to improve on the lower end?

So, the stated goals/objectives (as derived from the high-level business objectives) get crystallized with experience. It's a feedback loop.

How can you claim (or even aspire to) progression without indicators of progress?

As almost any arbitrary imposition would.

--
Gemaakt met Opera's revolutionaire e-mailprogramma:  
http://www.opera.com/mail/
(Remove the obvious prefix to reply.)

The book "Making Software" does a great job (if a bit voluminous and hard to read) of debunking pretty much all published metric-based "productivity" papers, and explaining why metrics aren't generally useful.

At least, that was my interpretation ;-)

Might help disabuse anyone tempted to believe too much in metrics...

Hope this helps, Best Regards, Dave

We have some pretty good metrics for measuring productivity. They are accurate and good info for future proposals.

The problem is management comes to the conclusion that we have learned from our mistakes and can therefore decrease the hours by 40% or so. It's usually displayed in the form that we should challenge ourselves and drive out inefficiencies with some song and dance meeting. So they underbid projects, basically throwing out the metrics. In reality, the original numbers turn out to be correct for new programs as new problems always arise that are different or unexpected then the previous project.

Sorry, I wasn't trying to infer that it was. Rather, I was trying to address Boudewijn's comment re: "How can a measurement of one metrics group affect the results of other metrics groups" This seemed like the most obvious way of making that point :>

It also doesn't help if your goal was, for example, to estimate resources required to *undertake* a particular project (before you've even gone to the trouble of preparing specications, etc.).

My point is that you need to understand *why* you want the metrics and what you expect to get *from* them before you can even address

*which* metrics are worth gathering and how to gather them! :-/

Yes, this gets to the issue I'm trying to address. Numbers for the sake of numbers are meaningless. You have to be able to understand what those numbers represent and how they impact other "issues". Presumably, all of this in the context of some "Grander Goal".

E.g., you can track how much toilet paper is consumed in an organization -- to the nearest whole *sheet*! But, that tells you nothing other than how much toilet paper remains in your store-room! :-/

So, if *that* is your goal, then you have a wonderful metric (note however, that it doesn't tell you when to order more or why you are using TP at this rate). Otherwise, you just have a "number"...

That's exactly the problem I'm addressing. The Developers aren't the ones asking for metrics. :> My claim is the "higher ups" (asking for the metrics) need to better understand what they want and why they want it. Else it is just "bean counting", literally... "beans" having no bearing on anything else in the organization or process!

[snip]

I agree. As I said before, I have no problem with metrics. Rather, my problem (and the issue I am trying to address, here) is with "metrics for the sake of metrics" -- without a reason and plan for their use.

I'm trying to come up with such "rational plans" so the "higher ups" can say, "No, *that's* not what we're after" or "Yes, exactly!". From *there*, you can begin to sort out *which* metrics are important and how to implement them.

Meaning I can't just buy a couple of copies and hand them to The Powers That Be :< Reading something "for themselves" somehow seems to be more credible than hearing someone else *distill* that same information for them...

Thanks! I'll pick up a copy and see if there are, at least, pertinent *sections* that I can pass on to be read -- if not the entire text itself.

So, in your case, they are used to measure past performance as a (hopeful) predictor of *future* performance (?)

How *early* in the planning/exploratory phase (for those future projects) do you draw on those numbers? Are these used to prepare estimates/plans for how long a project "will take"? Or, are they used to determine which (if any) projects are practical/profitable enough to undertake?

I.e., do "you" sit down with potential projects and use your metrics to rule out certain projects based on "costs" extrapolated from these metrics (if so, are you using anything other than a "gut feel" to gauge the complexity of the proposed projects?)? Or, once "someone" has decided to undertake a project, do you come up with a more detailed appraisal of what is involved *in* that project and *then* use the metrics to determine what sort of resources you need to make available to complete it in a particular timeframe/budget/etc.?

But that doesn't speak to the ills of the metrics (nor the means by which they are gathered). Rather, it speaks to an immaturity in the users of those data. I.e., it is very hard to compensate for "bad data". OTOH, it is easier (relatively speaking) to "fix" the problems of bad data *consumers* (get a new set of consumers! :-/ )

Specifically, which metrics do you track and, of those, which do you place most credence in?

Em 12/7/2011 01:45, Don Y escreveu:

Maybe yes, but them they'd have little business value as the use of metrics is intrinsically connect to the ability of _comparing_ them.

Maybe not, if you use metrics in right 'cut' of the technology and work them towards the business objectives. The complexity measures of algorithms would apply equally well in any of the above cited realms. If "only kept in a tree ring binder" then all the effort and resources to gather this data is obviously wasted, on the other hand if used to ascertain adequate coverage of tests, them they start to make business sense (at least for me ;-) ).

There is a phrase (incorrectly attributed to Deming) from P. Drucker ?If you can't measure it, you can't manage it.?

In fact, no! This is the first sin in our profession: to believe in this fallacy...

If loopholos cannot (even if approximately) be converted in dollars, then it would have no business value and the gathering of this information should be dropped from the organization. It would have the beneficial effect to save dollars!

All good so far.

Bad, bad... using metrics which can only be used within an organization hinders the comparison much needed to do business in the open world:

Buyer: "We would like to buy some lumber of you folks. How much do you charge per ton?" Seller: "No, no, no, sir! Here in this mill we use an internal metric we call knots of wood. How many knots are interested in?

The _values_ of the metrics obviously yes, but the nature of them, clearly a mistake taken for a lot of people without enough experience in metrics.

--
Cesar Rabak
GNU/Linux User 52247.
Get counted: http://counter.li.org/

Em 13/7/2011 15:02, Don Y escreveu: [snipped]

This is a symptom of a more fundamental problem: because in our profession we belittle use of metrics, we don't have the training to provide them on the time zero they're mentioned and asked by "higher ups".

People who work with much less predictable environments (say salespeople) are able to work with metrics scrutinizing their performance and do not complain nor one percent of what SW engineers complain about a metrics program for their roles.

'Beans' are the essence of the organization and the only way to choose between one or another process is by counting them!! Any other way of thinking is to be left to hobbyists or amateur work.

[snipped]

Depending on the size of the organization you're in, you may not have access to the 'layer' that uses the metrics, so just because you did not see them doesn't follow that the plan or rational use is not there.

Start by thinking what the things you would like to report and the ways you would like support your claims that 'it makes sense' to buy an expensive piece of HW or a software package...

When you arrive at a certain number (try to maintain this number in the five to nine range) look what the inputs would have to be gathered to produce those metrics and start a metrics program _at your level_.

--
Cesar Rabak
GNU/Linux User 52247.
Get counted: http://counter.li.org/

Em 13/7/2011 11:35, steve escreveu:

Yes this is a common misuse of all forms of estimation.

Also typical. With domino effect consequences.

And them create more attrition internally and with the clients that then require 'top notch performance account managers' or lawyers if the relationship goes sour.

--
Cesar Rabak
GNU/Linux User 52247.
Get counted: http://counter.li.org/

Em 13/7/2011 18:59, Don Y escreveu:

Yes. This is a common business use of metrics.

Depending on the sophistication of your metrics program, you can go as far as as soon as the opportunity had been detected.

Both uses are commonplace in mature organizations.

Notice that one of the ways of perceiving quality is the regularity a person/process performs in a task. Gathering the metrics and understanding the structural reasons for their variation makes the firm to rise its maturity.

You certainly can come with several things better than 'gut feel'.

If '"someone" "someone" has decided to undertake a project' already, without taking in account rules to obviate risks by the use of the historic data and other estimation tools, then obviously, you could arrive at a moot point.

OTOH, if the team who 'receives' the job to fulfill has the metrics, they would be in a better ground to negotiate internally and avoid pressure on them.

--
Cesar Rabak
GNU/Linux User 52247.
Get counted: http://counter.li.org/

I have no problem with metrics _in principal_.

_In fact_, however, I have not seen them applied well. The best 'software metrics' that I have ever seen applied weren't software-specific at all -- it was just good old GANTT chart "percent of task done", rolled up to a project completion metric.

But the metrics that bean counters want are things like lines of code written, adherence to complexity metrics, etc. In other words, they either want concrete 'measurable' signs of progress (without being willing to believe project schedules), which rushes the development and forces my team to turn out crap, or they want metrics that are easily gamed (and the people who are actually in charge don't want to hear about their metrics anyway).

Sales guys have much shorter time horizons than most embedded software engineers, much clearer goals, clearly defined measures of success ($$, either net or gross), and the advantage that you don't get to be a CEO unless you're at least something of a sales guy, and therefor understand the turf.

Yet the beans that are counted are often misleading, illusory, or flat out delusional. Which would you rather have: a product that costs $1M to develop, breaks in the field, alienates customers, and leads to lost sales for years, or a product that costs $5M to develop, works correctly and well straight out of the chute, and saves your marketing budget because your advertising becomes word-of-mouth?

Lines-o-code (or circuit boards laid out), prototype deadlines that allow the discarding 'cumbersome' quality controls, preproduction goals of being just in time for the trade show with no funding for actually making a product manufacturable -- these are the sorts of "metrics" that I've had the opportunity to work to. And they don't work!!

So, yea -- I'm kinda anti-metric in practice.

--

Tim Wescott
Wescott Design Services
http://www.wescottdesign.com

Do you need to implement control loops in software?
"Applied Control Theory for Embedded Systems" was written for you.
See details at http://www.wescottdesign.com/actfes/actfes.html

Yes, but perhaps you can numb them into retreat. "Read this book and you'll understand why this won't work, and why it may make you look foolish..."

Anyway, let us know what you think of the book ! Best Regards, Dave

That is a matter of education. The bean counters want something measurable, and stop thinking at the level of simple metrics that have little meaning. While a basic task dividing and tracking system already gives better beans to count.

Sounds like beans are not the only problem, you're also lacking in the requirements department.

--
Gemaakt met Opera's revolutionaire e-mailprogramma:  
http://www.opera.com/mail/
(Remove the obvious prefix to reply.)