Hi,

As per your own explanation, the web service calls 5 other web services. Let
us just take a look at what that means to your web server. Each web service
call is a HTTP request that has to be processed by IIS.  This request is
spawning 5 other HTTP requests and they are all running synchronously which
means one waits for the other to finish and blocks the thread till that is
done.

Now, you are saying that your test harness in VB is simulating multiple
users. Even if you did about 100 users - that means that you are calling 5 x
100 = 500 HTTP calls.

The ASP.NET worker process has a finite number of threads that it can spawn
(specified in machine.config) and further requests will just have to wait for
earlier threads to finish. And since your CPU utilization is probably running
at 100% by then, your IIS starts recycling the process. All this causes a lot
of HTTP failed requests and also the timeouts.

There are a few things you can do:
1. I don't know about your application, but surely making a Web Service call
five other services suggests to me that you have a design with too many
services. Since you have the other components that the Web Service wraps on
the same local system - why don't you reference them directly?  You can also
write some proxy code to find out if the components exist locally and
instantiate them directly if available - and if not, go make a web service
call.

2. If you are on ASP.NET 2.0, consider using Asynchronous Pages - a new
feature that spawns threads, but does not block the thread - you'll
definetely get an improvement - but depends on how much that will really help.

3. Use memory profilers (like the CLR profiler, Windows perf monitor) and
find out what piece of code is causing the delays / bottlenecks. In all
probability you will find that it is just because all your ASP.NET worker
threads are getting used up.

Hope that helps!

Regards
Pandurang

Check http://www.thejoyofcode.com/Tuning_the_ThreadPool.aspx for an excellent
article!

Regards,
Pandurang