Home | Contact Us | FAQ | Search & Site Map | Link to Us
Sign In | Join | Other 45 Sites in Network
HomeAnnouncementsFree MagazinesWhite PapersSubmit Content
Discussion GroupsASP.NETWindows FormsLanguages.NET FrameworkVisual Studio.NET
Articles.NET FrameworkASP.NETToolsWindows Forms
.NET DirectoryOpen Source ProjectsUser GroupsWeb Resources
Related Topics
Visual Basic 6SQL ServerMS AccessOther DB ProductsMS Server ProductsMore Topics ...

.NET Forum / .NET Framework / General / March 2008

Tip: Looking for answers? Try searching our database.

Unit Testing - The Merit of Writing Tests First is Questioned

Thread view: 
Enable EMail Alerts  Start New Thread
Thread rating: 
Robert Cramer - 20 Mar 2008 18:26 GMT
So I'm looking at this test-driven development (TDD) paradigm, and I like
the idea of having automated tests. That idea is not new.

But according to the TDD approach, we are to write our unit tests FIRST -
even before writing the code that the test is ultimately going to
test/verify.

I do see some merits of TDD - and doing the testing first - but what I think
is a practically guaranteed artifact of this approach is that many if not
most unit tests will become obsolete [and therefore have to be rewritten]
with any non trivial refactoring effort (which is also part of the whole TDD
paradigm). So, if I understand correctly, we are to write unit tests, then
write our code, then refactor the code almost immediately - and bam - just
like that, many of the unit test will also have to be rewritten or
refactored because the code being tested has been refactored [perhaps
refactored out of existance as we remove duplication, encapsulate stuff that
changes and recompose it with its original pre-refactoring class, etc].

Am I missing something?

Your thoughts and opinions are appreciated - specifically on the idea that
if I decide to write my unit tests first (per mainstream TDD), then many of
those tests won't be around [or will, themselves, need to be refactored] by
the time version 1of the system goes live.

Thanks!
Peter Duniho - 20 Mar 2008 18:58 GMT
> So I'm looking at this test-driven development (TDD) paradigm, and I like
> the idea of having automated tests. That idea is not new.
[quoted text clipped - 10 lines]
>
> Am I missing something?

Yes and no, I think.

First a caveat: I have worked in a variety of development environments,  
none of which use "TDD" per se, but at least some of which did rely on  
automated tests as a specific tool for ensuring code correctness.  So I  
can't really speak from the TDD perspective per se, but I at least know a  
little about the idea and have a reasonably broad range of experiences  
with varying approaches to testing.  :)

Anyway, I don't think your observation is wrong.  It's just that you seem  
to be implying that this sort of requirement for revision is a bad thing.

I think that whether you write the test first or later, there is always  
still the potential for it to need rewriting.  So it's not like you can  
ensure that you only ever have to write a test once.

One thing I like about the idea of writing the tests first is that they  
become part of the specification that documents how the actual code is  
supposed to work.  I've had too many experiences with vague  
specifications, or specifications that are demonstrably incorrect, often  
in the sense that they describe behavior that's impractical or  
impossible.  Developing a test case can often reveal these flaws before  
you get too far into the code design

The other thing is that you seem to be saying that a redesign of your code  
will always require a rewrite of the test.  This is only true if you take  
the test paradigm to its extreme limit (which is certainly possible, but  
not necessarily common).  If the automated tests focus only on the outer  
layers of the code, on those things described by an operational  
specification, then the only thing that should really require a test  
rewrite is if that specification itself changes.  A refactoring of the  
code that changes only the implementation but not the visible behavior of  
the code wouldn't require a new test.

In reality I think most projects will fall somewhere in the middle, in  
which you start with the top-level spec, but then intermediate specs wind  
up being created, either implicitly or explicitly as the code design gets  
worked out.  And you may in fact wind up writing test for those  
intermediate layers.  But the deeper you go into the code, where there's  
an increased possibility that some major refactoring might occur, the less  
likely it is IMHO you'll be bothering with a unit test for that part of  
the code.

In the end, it's not really an either/or sort of thing.  It's a matter of  
where you put your focus.  In the end, you wind up doing varying amounts  
of work in different parts of the project, and the main question is where  
do you find it most valuable to invest the most effort?  IMHO there are no  
right or wrong answers here, but rather answers that fit the developer or  
team best according to their own needs, as well as their own strengths and  
weaknesses.

So, sure...you might wind up rewriting your tests because something  
changed.  But so what?  If that means you spend less time on something  
else, and especially if it means that the rest of the code is much more  
likely to be correct, that seems like a fair trade-off.

The trick is to make sure you're really getting the benefit of that  
trade-off by writing good tests.  Garbage in, garbage out.  :)

Pete
Cowboy (Gregory A. Beamer) - 21 Mar 2008 16:45 GMT
> So I'm looking at this test-driven development (TDD) paradigm, and I like
> the idea of having automated tests. That idea is not new.
>
> But according to the TDD approach, we are to write our unit tests FIRST -
> even before writing the code that the test is ultimately going to
> test/verify.

This is the purist approach and it has both merits and detractors. Most of
the detractors are paradigm shifts rather than actual impediments.

> I do see some merits of TDD - and doing the testing first - but what I
> think is a practically guaranteed artifact of this approach is that many
> if not most unit tests will become obsolete [and therefore have to be
> rewritten] with any non trivial refactoring effort (which is also part of
> the whole TDD paradigm).

If you do a code first, design second approach, this is a risk. If you take
time to brainstorm your application and figure out use cases, flows, etc.,
there is very little danger. Organization is the key, but then again, you
should not be building an application that you do not understand.
Unfortunately, this is VERY common in the industry, as managers try to limit
design time, throw out specs that are worthless, etc.

BTW, refactoring, for the most part, does not break tests, as the majority
of it is either a) refactoring out duplicate code or b) refining current
code. If you are changing interfaces, etc., you are not really refactoring,
you are altering. Let's take an example:

if(x=1) {}
elseif (x=2) {}
elseif (x=3) {}
elseif (x=4) {}
else {}

Each of the conditions has to be walked, which is inefficient. A common
refactor would be to change to a switch (Select ... Case) in VB.

switch(x)
{
   case 1: {}
   case 2: {}
   case 3: {}
   case 4: {}
   default: {}
}

This creates a state machine. That is a refactor. But, note, that there is
nothing happening here that changes my public interface, so the test still
works.

In fact, since most refactoring is trying to improve, not alter, code, NOT
having tests is a major detriment, as your tests, when written properly,
protect you against new bugs in old code.

Does refactoring ever break tests? Certainly. But, more often than not, it
is because your refactoring revealed a flaw in your test. Either you were
running the test for the wrong behavior or you have set up an overly
optimistic test, or similar.

> So, if I understand correctly, we are to write unit tests, then write our
> code, then refactor the code almost immediately - and bam - just like
[quoted text clipped - 4 lines]
>
> Am I missing something?

Yes. You are missing the core of TDD.

1. Design - Determine what you are building
2. Decide - Agree on functionality to be built
3. Divide - Break work down into bite sized chunks
4. Write Tests - write a test on the expected behavior
5. Write code to pass the test - red to green
6. Refactor - remove any "code smells" (repeat code being the most common)

If you start with step 4, you will probably end up rewriting your tests as
you really have no clue what you are building. In that case, return to step
one. I can even add a pre-step, which is brainstorm.

If you want to get a good idea of how to start design for customer facing
apps (and I include internally facing apps here), you can watch the free
user experience videos at http://sessions.visitmix.com (need to have
SIlverlight installed). Some really nice sessions on design are UX04, UX05,
UX06 and UX07 (in that order), which are videos from Adaptive Path. The
middle of each is boring, as people are doing exercises, so fast forward to
the last 10-15 minutes when they start that.

Another thing to consider is unit tests are just that. They test a single
routine. You can expand this a bit, but if you are system testing, your
tests are too broad.

Let's take an example, which you might be thinking fits your "problem" with
TDD and why it does not.

You have a test that tests a User object. In your application, you will pull
the information from the database and diplay on a web page. Something like:

ASP.NET page
Welcome <asp:Label id="nameLabel" runat="server"/>

Code behind
nameLabel.Text = user.UserName;

You have a user object that is filled by some facade method. A quick stub
would be something like:

public static User GetUserFromId(string userID)
{
   //Stuff to test id and pull user object from database here
   return user;
}

So, you set up the following test (using Visual Studio unit test format):

[TestMethod]
public void TestGetUserFromIdSuccess()
{
   string userID = "G5F765";
   string expectedName = "Greg Beamer";

   User actual = UserFactory.GetUserFromId(userID);

   Assert.AreEqual(expectedName, actual.Name, "Name is different");
}

Suppose, however, that the test data no longer contains the same name for
that user. It fails. But, you are not supposed to be testing the database
retrieval at the business layer. You should be injecting the dependency, so
it always returns the right name. This may sound like "cheating", but you
are only testing the code in THAT routine.

Now, if you change the entire business layer and how you retrieve data, you
will break the application. But that is more likely to happen if you have
not designed it up front, at least in most instances.

> Your thoughts and opinions are appreciated - specifically on the idea that
> if I decide to write my unit tests first (per mainstream TDD), then many
> of those tests won't be around [or will, themselves, need to be
> refactored] by the time version 1of the system goes live.

I disagree. In fact, I have used TDD for years now and find that 90%+ of my
tests are still around. There are a few that have changed, but if you design
first and then code, you find that you eliminate a lot of useless rabbit
holes before you even start coding.

Signature

Gregory A. Beamer
MVP, MCP: +I, SE, SD, DBA

Subscribe to my blog
http://gregorybeamer.spaces.live.com/lists/feed.rss

or just read it:
http://gregorybeamer.spaces.live.com/

*************************************************

| Think outside the box!

*************************************************
Robert Cramer - 21 Mar 2008 22:03 GMT
Thank you Gregory and Peter for your thoughtful and helpful responses.

RE:
<<
  1. Design - Determine what you are building
  2. Decide - Agree on functionality to be built
  3. Divide - Break work down into bite sized chunks
  4. Write Tests - write a test on the expected behavior
  5. Write code to pass the test - red to green
  6. Refactor - remove any "code smells" (repeat code being the most
common)

  If you start with step 4, you will probably end up rewriting your tests
as you really have no clue what you are building.

This is exactly my concern [starting with step 4]. I have been developing
non trivial business apps professionally for more than 12 years (complete
lifecycle) and recently attended a TDD seminar in which the unit test was
presented as driving everything. When questions were raised about design
(step 1, according to you -- and I agree with you), that was downplayed with
the explanation that the unit tests will inform your design decisions. Yes -
the creation of unit tests are to inform our design decisions according to
the guys leading the session. In fact, according to the presenters, that's
the "driven" part of test-driven development" -- the tests DRIVE the
Development. That's where they lost credibility with me. But I do like the
idea of having automated tests so that I can periodically run the tests to
quickly verify that life is good with the system after modifications have
been made. But we don't need TDD in order to have automated tests. And I
prefer to drive my designs from, well, a good focused design effort (not
some suite of tests). So rather than dismissing TDD outright, I decided that
the leaders of the sessions were confused, incompetent, or both. So I'm
still wanting to know the value of TDD. The resources I've found seem to
center on the "almighty unit test" as if that should drive everything else -
rather than the automated set of unit tests being part of a more
comprehensive and standard development cycle (as promoted, for example in
the book, Code Complete).

Thank you again for your thoughtful responses. I'll try to take another
serious look at TDD - or at least borrow some ideas from it even if I don't
fully embrace it.

-RC
Jon Skeet [C# MVP] - 21 Mar 2008 22:42 GMT
> This is exactly my concern [starting with step 4]. I have been developing
> non trivial business apps professionally for more than 12 years (complete
[quoted text clipped - 4 lines]
> the creation of unit tests are to inform our design decisions according to
> the guys leading the session.

I'd agree with that, actually.

Tests shouldn't necessarily drive *architecture* but I find it's
reasonable that they drive *design*.

There's a simple reason here - it generally means you end up with a
class which is easy to use as well as easy to write. You're immediately
thinking from the *caller's* point of view instead of from an
implementation side. You think "what do I want to do with this class"
rather than "what do I want this class to provide".

I know that sounds like a subtle distinction, but I've found it really
makes a big difference. Classes which have been designed by writing
tests *tend* to be easier to use later on. They also *tend* to
naturally express their dependencies in nice ways (e.g. interfaces).

That's just in my limited experience though.

Signature

Jon Skeet - <skeet@pobox.com>
http://www.pobox.com/~skeet   Blog: http://www.msmvps.com/jon.skeet
World class .NET training in the UK: http://iterativetraining.co.uk

Robert Cramer - 22 Mar 2008 03:20 GMT
Responses inline:

<snip>

> Tests shouldn't necessarily drive *architecture* but I find it's
> reasonable that they drive *design*.
[quoted text clipped - 4 lines]
> implementation side. You think "what do I want to do with this class"
> rather than "what do I want this class to provide".

> I know that sounds like a subtle distinction, but I've found it really
> makes a big difference.

Interesting - I can easily see how that only *sounds* subtle, but I can
easily recognize how changing the perspective ([what can I do with this
class?] as opposed to [what can this class do?]) could result in some very
different, perhaps better, design decisions.

I was thinking architecture all along. My experience is that the term
"architecture" is so heavily overloaded or outright misused that I
completely missed the distinction between design and architecture. Great
catch.

>Classes which have been designed by writing
> tests *tend* to be easier to use later on. They also *tend* to
> naturally express their dependencies in nice ways (e.g. interfaces).

Okay, that's believeable, and makes TDD worth a very close 2nd look on my
part. I guess I need to really get past the presentation.

> That's just in my limited experience though.

I appreciate your input. Do you have any suggested readings? Yes, I know I
could Google this, and I have - but I'd like a recommendation from someone I
respect.

-RC
Jon Skeet [C# MVP] - 22 Mar 2008 08:52 GMT
> > I know that sounds like a subtle distinction, but I've found it really
> > makes a big difference.
[quoted text clipped - 3 lines]
> class?] as opposed to [what can this class do?]) could result in some very
> different, perhaps better, design decisions.

That's handy, because I suspect I wouldn't be able to really explain it
if you didn't "get it" so easily :)

> I was thinking architecture all along. My experience is that the term
> "architecture" is so heavily overloaded or outright misused that I
> completely missed the distinction between design and architecture. Great
> catch.

No problem. Of course, due to that overloading it's quite possible that
my understanding of the words "architecture" and "design" aren't the
same as yours either!

> >Classes which have been designed by writing
> > tests *tend* to be easier to use later on. They also *tend* to
[quoted text clipped - 8 lines]
> could Google this, and I have - but I'd like a recommendation from someone I
> respect.

Well, I haven't read it since the release (must get round to it some
time) but "Test-Driven" (http://www.manning.com/koskela/) was excellent
when I read a preview.

Unfortunately it's now been a few years since I first learned about
TDD, so I can't remember what the process was like... I do remember
that learning about mocking was a lightbulb moment. Otherwise
dependencies kill you...

Signature

Jon Skeet - <skeet@pobox.com>
http://www.pobox.com/~skeet   Blog: http://www.msmvps.com/jon.skeet
World class .NET training in the UK: http://iterativetraining.co.uk

Lasse Vågsæther Karlsen - 22 Mar 2008 13:14 GMT
>>> I know that sounds like a subtle distinction, but I've found it really
>>> makes a big difference.
[quoted text clipped - 5 lines]
> That's handy, because I suspect I wouldn't be able to really explain it
> if you didn't "get it" so easily :)

I've had a similar discussion recently and the best example a collegue
came up with was: take a bike, and describe how it works, then... as an
alternative, take the cyclist, and describe what he wants to do.

Turns out there are many solutions to the cyclists problem, and it's
actually very hard to describe a bike to someone that hasn't seen it. In
other words, the chances you'll make a bike for someone who doesn't know
what a bike is, and actually solve his problem, is going to be smaller
than if you ask the cyclist first, what do you want to do and how do you
want to do it, and design the bike (or whatever) to fit those needs and
usage patterns.

>> I was thinking architecture all along. My experience is that the term
>> "architecture" is so heavily overloaded or outright misused that I
[quoted text clipped - 10 lines]
>> Okay, that's believeable, and makes TDD worth a very close 2nd look on my
>> part. I guess I need to really get past the presentation.

I can second that. I have rewritten one big class for a pet project
three times and the last time I did the time to actually write about 50
tests first that demoed how I would use it, and just hardcode my way
through all the tests to that they compiled and ran. The final resulting
class was way different than the first two iterations, much easier to
use and turned out to be easier to do internally as well. Just changing
the mindset from the API creater (or is that *creator*?) to the API user
made quite a difference.

<snip>

Signature

Lasse Vågsæther Karlsen
mailto:lasse@vkarlsen.no
http://presentationmode.blogspot.com/
PGP KeyID: 0xBCDEA2E3

news.microsoft.com - 22 Mar 2008 12:30 GMT
I worked with Thoughtworks on a project for a year, and though I grokked
TDD, I think that this: http://dannorth.net/introducing-bdd is a much better
way of thinking about it.
Arne Vajhøj - 24 Mar 2008 02:49 GMT
> So I'm looking at this test-driven development (TDD) paradigm, and I like
> the idea of having automated tests. That idea is not new.
>
> But according to the TDD approach, we are to write our unit tests FIRST -
> even before writing the code that the test is ultimately going to
> test/verify.

Correct. Or written by someone other than the person writing the code.

Otherwise:
  implementation problems => tests => design
for TDD or:
  tests will tent to test what the code does instead what it should do
for more traditional unit tests.

> I do see some merits of TDD - and doing the testing first - but what I think
> is a practically guaranteed artifact of this approach is that many if not
[quoted text clipped - 6 lines]
> refactored out of existance as we remove duplication, encapsulate stuff that
> changes and recompose it with its original pre-refactoring class, etc].

A refactoring in the low level sense should not require unit tests to
be rewritten.

A change in design (which you can call refactoring at the high level)
will require a change in unit tests.

It should not happen that often.

And if it does happen, then I would expect rewriting unit tests
to be a very small fraction of the work added.

I don't see it as a problem.

Arne

PS: I am not particular TDD oriented. I see the main advantage of
    extensive unit tests to show up after the code is developed
    and enter maintenance mode.

Free Magazines

Get these publications absolutely FREE for up to 12 months. There are no hidden fees and no obligation. Simply choose a title, complete the application form and submit it. Read more ...

Oracle MagazineNetwork ComputingComputer WorldBio-IT WorldeWeekInformation WeekInfosecurity
 
Sign In
Join
My Latest Posts
My Monitored Threads
My Blog
My Photo Gallery
My Profile
My Homepage

Start New Thread
Enable EMail Alerts
Rate this Thread



©2008 Advenet LLC   Privacy Policy - Terms of Use
This website includes both content owned or controlled by Advenet as well as content owned or controlled by third parties.