.NET Forum / .NET Framework / General / March 2008
Unit Testing - The Merit of Writing Tests First is Questioned
|
|
Thread rating:  |
Robert Cramer - 20 Mar 2008 18:26 GMT So I'm looking at this test-driven development (TDD) paradigm, and I like the idea of having automated tests. That idea is not new.
But according to the TDD approach, we are to write our unit tests FIRST - even before writing the code that the test is ultimately going to test/verify.
I do see some merits of TDD - and doing the testing first - but what I think is a practically guaranteed artifact of this approach is that many if not most unit tests will become obsolete [and therefore have to be rewritten] with any non trivial refactoring effort (which is also part of the whole TDD paradigm). So, if I understand correctly, we are to write unit tests, then write our code, then refactor the code almost immediately - and bam - just like that, many of the unit test will also have to be rewritten or refactored because the code being tested has been refactored [perhaps refactored out of existance as we remove duplication, encapsulate stuff that changes and recompose it with its original pre-refactoring class, etc].
Am I missing something?
Your thoughts and opinions are appreciated - specifically on the idea that if I decide to write my unit tests first (per mainstream TDD), then many of those tests won't be around [or will, themselves, need to be refactored] by the time version 1of the system goes live.
Thanks!
Peter Duniho - 20 Mar 2008 18:58 GMT > So I'm looking at this test-driven development (TDD) paradigm, and I like > the idea of having automated tests. That idea is not new. [quoted text clipped - 10 lines] > > Am I missing something? Yes and no, I think.
First a caveat: I have worked in a variety of development environments, none of which use "TDD" per se, but at least some of which did rely on automated tests as a specific tool for ensuring code correctness. So I can't really speak from the TDD perspective per se, but I at least know a little about the idea and have a reasonably broad range of experiences with varying approaches to testing. :)
Anyway, I don't think your observation is wrong. It's just that you seem to be implying that this sort of requirement for revision is a bad thing.
I think that whether you write the test first or later, there is always still the potential for it to need rewriting. So it's not like you can ensure that you only ever have to write a test once.
One thing I like about the idea of writing the tests first is that they become part of the specification that documents how the actual code is supposed to work. I've had too many experiences with vague specifications, or specifications that are demonstrably incorrect, often in the sense that they describe behavior that's impractical or impossible. Developing a test case can often reveal these flaws before you get too far into the code design
The other thing is that you seem to be saying that a redesign of your code will always require a rewrite of the test. This is only true if you take the test paradigm to its extreme limit (which is certainly possible, but not necessarily common). If the automated tests focus only on the outer layers of the code, on those things described by an operational specification, then the only thing that should really require a test rewrite is if that specification itself changes. A refactoring of the code that changes only the implementation but not the visible behavior of the code wouldn't require a new test.
In reality I think most projects will fall somewhere in the middle, in which you start with the top-level spec, but then intermediate specs wind up being created, either implicitly or explicitly as the code design gets worked out. And you may in fact wind up writing test for those intermediate layers. But the deeper you go into the code, where there's an increased possibility that some major refactoring might occur, the less likely it is IMHO you'll be bothering with a unit test for that part of the code.
In the end, it's not really an either/or sort of thing. It's a matter of where you put your focus. In the end, you wind up doing varying amounts of work in different parts of the project, and the main question is where do you find it most valuable to invest the most effort? IMHO there are no right or wrong answers here, but rather answers that fit the developer or team best according to their own needs, as well as their own strengths and weaknesses.
So, sure...you might wind up rewriting your tests because something changed. But so what? If that means you spend less time on something else, and especially if it means that the rest of the code is much more likely to be correct, that seems like a fair trade-off.
The trick is to make sure you're really getting the benefit of that trade-off by writing good tests. Garbage in, garbage out. :)
Pete
Cowboy (Gregory A. Beamer) - 21 Mar 2008 16:45 GMT > So I'm looking at this test-driven development (TDD) paradigm, and I like > the idea of having automated tests. That idea is not new. > > But according to the TDD approach, we are to write our unit tests FIRST - > even before writing the code that the test is ultimately going to > test/verify. This is the purist approach and it has both merits and detractors. Most of the detractors are paradigm shifts rather than actual impediments.
> I do see some merits of TDD - and doing the testing first - but what I > think is a practically guaranteed artifact of this approach is that many > if not most unit tests will become obsolete [and therefore have to be > rewritten] with any non trivial refactoring effort (which is also part of > the whole TDD paradigm). If you do a code first, design second approach, this is a risk. If you take time to brainstorm your application and figure out use cases, flows, etc., there is very little danger. Organization is the key, but then again, you should not be building an application that you do not understand. Unfortunately, this is VERY common in the industry, as managers try to limit design time, throw out specs that are worthless, etc.
BTW, refactoring, for the most part, does not break tests, as the majority of it is either a) refactoring out duplicate code or b) refining current code. If you are changing interfaces, etc., you are not really refactoring, you are altering. Let's take an example:
if(x=1) {} elseif (x=2) {} elseif (x=3) {} elseif (x=4) {} else {}
Each of the conditions has to be walked, which is inefficient. A common refactor would be to change to a switch (Select ... Case) in VB.
switch(x) { case 1: {} case 2: {} case 3: {} case 4: {} default: {} }
This creates a state machine. That is a refactor. But, note, that there is nothing happening here that changes my public interface, so the test still works.
In fact, since most refactoring is trying to improve, not alter, code, NOT having tests is a major detriment, as your tests, when written properly, protect you against new bugs in old code.
Does refactoring ever break tests? Certainly. But, more often than not, it is because your refactoring revealed a flaw in your test. Either you were running the test for the wrong behavior or you have set up an overly optimistic test, or similar.
> So, if I understand correctly, we are to write unit tests, then write our > code, then refactor the code almost immediately - and bam - just like [quoted text clipped - 4 lines] > > Am I missing something? Yes. You are missing the core of TDD.
1. Design - Determine what you are building 2. Decide - Agree on functionality to be built 3. Divide - Break work down into bite sized chunks 4. Write Tests - write a test on the expected behavior 5. Write code to pass the test - red to green 6. Refactor - remove any "code smells" (repeat code being the most common)
If you start with step 4, you will probably end up rewriting your tests as you really have no clue what you are building. In that case, return to step one. I can even add a pre-step, which is brainstorm.
If you want to get a good idea of how to start design for customer facing apps (and I include internally facing apps here), you can watch the free user experience videos at http://sessions.visitmix.com (need to have SIlverlight installed). Some really nice sessions on design are UX04, UX05, UX06 and UX07 (in that order), which are videos from Adaptive Path. The middle of each is boring, as people are doing exercises, so fast forward to the last 10-15 minutes when they start that.
Another thing to consider is unit tests are just that. They test a single routine. You can expand this a bit, but if you are system testing, your tests are too broad.
Let's take an example, which you might be thinking fits your "problem" with TDD and why it does not.
You have a test that tests a User object. In your application, you will pull the information from the database and diplay on a web page. Something like:
ASP.NET page Welcome <asp:Label id="nameLabel" runat="server"/>
Code behind nameLabel.Text = user.UserName;
You have a user object that is filled by some facade method. A quick stub would be something like:
public static User GetUserFromId(string userID) { //Stuff to test id and pull user object from database here return user; }
So, you set up the following test (using Visual Studio unit test format):
[TestMethod] public void TestGetUserFromIdSuccess() { string userID = "G5F765"; string expectedName = "Greg Beamer";
User actual = UserFactory.GetUserFromId(userID);
Assert.AreEqual(expectedName, actual.Name, "Name is different"); }
Suppose, however, that the test data no longer contains the same name for that user. It fails. But, you are not supposed to be testing the database retrieval at the business layer. You should be injecting the dependency, so it always returns the right name. This may sound like "cheating", but you are only testing the code in THAT routine.
Now, if you change the entire business layer and how you retrieve data, you will break the application. But that is more likely to happen if you have not designed it up front, at least in most instances.
> Your thoughts and opinions are appreciated - specifically on the idea that > if I decide to write my unit tests first (per mainstream TDD), then many > of those tests won't be around [or will, themselves, need to be > refactored] by the time version 1of the system goes live. I disagree. In fact, I have used TDD for years now and find that 90%+ of my tests are still around. There are a few that have changed, but if you design first and then code, you find that you eliminate a lot of useless rabbit holes before you even start coding.
 Signature Gregory A. Beamer MVP, MCP: +I, SE, SD, DBA
Subscribe to my blog http://gregorybeamer.spaces.live.com/lists/feed.rss
or just read it: http://gregorybeamer.spaces.live.com/
*************************************************
| Think outside the box! *************************************************
Robert Cramer - 21 Mar 2008 22:03 GMT Thank you Gregory and Peter for your thoughtful and helpful responses.
RE: << 1. Design - Determine what you are building 2. Decide - Agree on functionality to be built 3. Divide - Break work down into bite sized chunks 4. Write Tests - write a test on the expected behavior 5. Write code to pass the test - red to green 6. Refactor - remove any "code smells" (repeat code being the most common)
If you start with step 4, you will probably end up rewriting your tests as you really have no clue what you are building.
This is exactly my concern [starting with step 4]. I have been developing non trivial business apps professionally for more than 12 years (complete lifecycle) and recently attended a TDD seminar in which the unit test was presented as driving everything. When questions were raised about design (step 1, according to you -- and I agree with you), that was downplayed with the explanation that the unit tests will inform your design decisions. Yes - the creation of unit tests are to inform our design decisions according to the guys leading the session. In fact, according to the presenters, that's the "driven" part of test-driven development" -- the tests DRIVE the Development. That's where they lost credibility with me. But I do like the idea of having automated tests so that I can periodically run the tests to quickly verify that life is good with the system after modifications have been made. But we don't need TDD in order to have automated tests. And I prefer to drive my designs from, well, a good focused design effort (not some suite of tests). So rather than dismissing TDD outright, I decided that the leaders of the sessions were confused, incompetent, or both. So I'm still wanting to know the value of TDD. The resources I've found seem to center on the "almighty unit test" as if that should drive everything else - rather than the automated set of unit tests being part of a more comprehensive and standard development cycle (as promoted, for example in the book, Code Complete).
Thank you again for your thoughtful responses. I'll try to take another serious look at TDD - or at least borrow some ideas from it even if I don't fully embrace it.
-RC
Jon Skeet [C# MVP] - 21 Mar 2008 22:42 GMT > This is exactly my concern [starting with step 4]. I have been developing > non trivial business apps professionally for more than 12 years (complete [quoted text clipped - 4 lines] > the creation of unit tests are to inform our design decisions according to > the guys leading the session. I'd agree with that, actually.
Tests shouldn't necessarily drive *architecture* but I find it's reasonable that they drive *design*.
There's a simple reason here - it generally means you end up with a class which is easy to use as well as easy to write. You're immediately thinking from the *caller's* point of view instead of from an implementation side. You think "what do I want to do with this class" rather than "what do I want this class to provide".
I know that sounds like a subtle distinction, but I've found it really makes a big difference. Classes which have been designed by writing tests *tend* to be easier to use later on. They also *tend* to naturally express their dependencies in nice ways (e.g. interfaces).
That's just in my limited experience though.
 Signature Jon Skeet - <skeet@pobox.com> http://www.pobox.com/~skeet Blog: http://www.msmvps.com/jon.skeet World class .NET training in the UK: http://iterativetraining.co.uk
Robert Cramer - 22 Mar 2008 03:20 GMT Responses inline:
<snip>
> Tests shouldn't necessarily drive *architecture* but I find it's > reasonable that they drive *design*. [quoted text clipped - 4 lines] > implementation side. You think "what do I want to do with this class" > rather than "what do I want this class to provide".
> I know that sounds like a subtle distinction, but I've found it really > makes a big difference. Interesting - I can easily see how that only *sounds* subtle, but I can easily recognize how changing the perspective ([what can I do with this class?] as opposed to [what can this class do?]) could result in some very different, perhaps better, design decisions.
I was thinking architecture all along. My experience is that the term "architecture" is so heavily overloaded or outright misused that I completely missed the distinction between design and architecture. Great catch.
>Classes which have been designed by writing > tests *tend* to be easier to use later on. They also *tend* to > naturally express their dependencies in nice ways (e.g. interfaces). Okay, that's believeable, and makes TDD worth a very close 2nd look on my part. I guess I need to really get past the presentation.
> That's just in my limited experience though. I appreciate your input. Do you have any suggested readings? Yes, I know I could Google this, and I have - but I'd like a recommendation from someone I respect.
-RC
Jon Skeet [C# MVP] - 22 Mar 2008 08:52 GMT > > I know that sounds like a subtle distinction, but I've found it really > > makes a big difference. [quoted text clipped - 3 lines] > class?] as opposed to [what can this class do?]) could result in some very > different, perhaps better, design decisions. That's handy, because I suspect I wouldn't be able to really explain it if you didn't "get it" so easily :)
> I was thinking architecture all along. My experience is that the term > "architecture" is so heavily overloaded or outright misused that I > completely missed the distinction between design and architecture. Great > catch. No problem. Of course, due to that overloading it's quite possible that my understanding of the words "architecture" and "design" aren't the same as yours either!
> >Classes which have been designed by writing > > tests *tend* to be easier to use later on. They also *tend* to [quoted text clipped - 8 lines] > could Google this, and I have - but I'd like a recommendation from someone I > respect. Well, I haven't read it since the release (must get round to it some time) but "Test-Driven" (http://www.manning.com/koskela/) was excellent when I read a preview.
Unfortunately it's now been a few years since I first learned about TDD, so I can't remember what the process was like... I do remember that learning about mocking was a lightbulb moment. Otherwise dependencies kill you...
 Signature Jon Skeet - <skeet@pobox.com> http://www.pobox.com/~skeet Blog: http://www.msmvps.com/jon.skeet World class .NET training in the UK: http://iterativetraining.co.uk
Lasse Vågsæther Karlsen - 22 Mar 2008 13:14 GMT >>> I know that sounds like a subtle distinction, but I've found it really >>> makes a big difference. [quoted text clipped - 5 lines] > That's handy, because I suspect I wouldn't be able to really explain it > if you didn't "get it" so easily :) I've had a similar discussion recently and the best example a collegue came up with was: take a bike, and describe how it works, then... as an alternative, take the cyclist, and describe what he wants to do.
Turns out there are many solutions to the cyclists problem, and it's actually very hard to describe a bike to someone that hasn't seen it. In other words, the chances you'll make a bike for someone who doesn't know what a bike is, and actually solve his problem, is going to be smaller than if you ask the cyclist first, what do you want to do and how do you want to do it, and design the bike (or whatever) to fit those needs and usage patterns.
>> I was thinking architecture all along. My experience is that the term >> "architecture" is so heavily overloaded or outright misused that I [quoted text clipped - 10 lines] >> Okay, that's believeable, and makes TDD worth a very close 2nd look on my >> part. I guess I need to really get past the presentation. I can second that. I have rewritten one big class for a pet project three times and the last time I did the time to actually write about 50 tests first that demoed how I would use it, and just hardcode my way through all the tests to that they compiled and ran. The final resulting class was way different than the first two iterations, much easier to use and turned out to be easier to do internally as well. Just changing the mindset from the API creater (or is that *creator*?) to the API user made quite a difference.
<snip>
 Signature Lasse Vågsæther Karlsen mailto:lasse@vkarlsen.no http://presentationmode.blogspot.com/ PGP KeyID: 0xBCDEA2E3
news.microsoft.com - 22 Mar 2008 12:30 GMT I worked with Thoughtworks on a project for a year, and though I grokked TDD, I think that this: http://dannorth.net/introducing-bdd is a much better way of thinking about it.
Arne Vajhøj - 24 Mar 2008 02:49 GMT > So I'm looking at this test-driven development (TDD) paradigm, and I like > the idea of having automated tests. That idea is not new. > > But according to the TDD approach, we are to write our unit tests FIRST - > even before writing the code that the test is ultimately going to > test/verify. Correct. Or written by someone other than the person writing the code.
Otherwise: implementation problems => tests => design for TDD or: tests will tent to test what the code does instead what it should do for more traditional unit tests.
> I do see some merits of TDD - and doing the testing first - but what I think > is a practically guaranteed artifact of this approach is that many if not [quoted text clipped - 6 lines] > refactored out of existance as we remove duplication, encapsulate stuff that > changes and recompose it with its original pre-refactoring class, etc]. A refactoring in the low level sense should not require unit tests to be rewritten.
A change in design (which you can call refactoring at the high level) will require a change in unit tests.
It should not happen that often.
And if it does happen, then I would expect rewriting unit tests to be a very small fraction of the work added.
I don't see it as a problem.
Arne
PS: I am not particular TDD oriented. I see the main advantage of extensive unit tests to show up after the code is developed and enter maintenance mode.
Free MagazinesGet these publications absolutely FREE for up to 12 months. There are no hidden fees and no obligation. Simply choose a title, complete the application form and submit it. Read more ...
|
|
|