Comments on: Is Code Coverage Really All That Useful?

By: Mark Utting

Mark Utting — Tue, 09 Dec 2008 20:59:34 +0000

As Mathias said (10/29/2008): "a good test suite should be sensitive and reflect tightly your requirements/assumptions; therefore, if you mutated your code, your tests should fail. So you could measure your tests quality by mutating the code, and looking at how much failure you create."

I like to use mutation testing to measure the effectiveness/quality of my JUnit test suites.
It is more demanding than code coverage…

Jumble (http://jumble.sourceforge.net) is a fast and effective tool that gives you a 0..100% score for each of your Java source code classes, by applying one or two simple mutations to each command and expression in the class and seeing if your tests are strong enough to detect the mutation. It works by mutating the bytecode directly.

The current release of Jumble (June 2007) works with JUnit 3.8.1, but the SVN repository version works with JUnit 4.4, as will the real-soon-now release.

By: Jason

Jason — Wed, 05 Nov 2008 07:41:24 +0000

I don’t think any rational person would see code coverage as the be-all-end-all or silver bullet. It’s a great metric, but only 1 metric of MANY to determine quality. CI, functional and system testing needs to happen in conjunction with unit testing and those unit tests should be code reviewed by humans to make sure they are useful tests. 100% code coverage is useless if the tests suck.

By: André Cardoso

André Cardoso — Wed, 05 Nov 2008 04:02:38 +0000

I remember from university that there are other metrics related to data paths/data flow, that could be useful for measuring if the corner cases are "covered" (boundary tests, data equivalence partitions, loops, etc).

By: Thomas

Thomas — Wed, 05 Nov 2008 01:56:38 +0000

For really serious code thrashing, try mutation testing (ref: Wikipedia). For Ruby there’s heckle http://github.com/ryansobol/heckle , for Java there’s Jester (though that doesn’t look very current). Even if you have 100% coverage, mutating the code can help you find unexpected weirdness.

By: lava kafle

lava kafle — Tue, 04 Nov 2008 21:46:43 +0000

hey a new approach but we need to use coverage tools to get rid of Miss counts so we do not miss optimal path essential for covering certain codes duri9ng testing. We need to get experience from rigorous manual testing first to use code coverage effectively.

By: Kevin Pang

Kevin Pang — Tue, 04 Nov 2008 16:11:45 +0000

@Comment

Not rude at all. In fact, I would agree with you if only for one thing: the strawman isn’t really a strawman — to me at least — since I’ve had to deal with customers and management who have said exactly that.

If it seems like I’m arguing against a silly assumption, it’s because I am and it is. Unfortunately, it’s an argument that needs to be made because a lot of developers and managers out there *do* see code coverage as silver bullet. Maybe I’m preaching to the choir.

By: Comment

Comment — Tue, 04 Nov 2008 15:48:47 +0000

Not to be rude in any way, and haven’t read all the comments on your page, but you are beating a straw man here.

Opponents assumed argument:
100% code coverage is the the reliable measurement.

Your argument:
Proof that 100% line coverage does not fullfill above stated argument.

Conclusion:
Code coverage is not useful

Of course tests are useful, it’s a matter of using them right. To travesty our analogy: the engineer should run a ten ton truck with run down tires over the bridge intended to be tested.

Your comment is, truth to be said, a bit naive, and it seems that you are a bit irritated with the fanboys of code coverage.

Testing is of course useful.

No silver bullet.

By: David Bullock

David Bullock — Tue, 04 Nov 2008 15:34:41 +0000

Will people ever stop asking meaningless questions to bait unsuspecting readers into reading whatever happened to enter the author’s mind one afternoon?

By: Chris Nash

Chris Nash — Tue, 04 Nov 2008 15:27:00 +0000

Agree with the main point here – all code coverage tells you for certain is how much of your code has *never* been run, not how much actually works. I’ve worked on projects that have bragged about 90% code coverage – I’d much rather people be more honest, and say there’s 10% out there we know nothing at all about.

By: Craig D

Craig D — Tue, 04 Nov 2008 14:38:00 +0000

I’ve seen — and admittedly created — enough lame unit tests to realize that code coverage is a useless measure. Unit testing should be about quality, not quantity. Testing needs to cover boundry cases and use random inputs when possible to be of any use in catching errors that a BDEU (brain dead end user) can easily invoke.