Hello,

I develop a serverside application which should connect to external systems
through Http to get a lot of data (megs of Xml per request). I use
HttpWebRequest. Each http request is sent in a separate thread in order to
not lock the application. The application should serve multiple clients and
it must be able to send multiple Http requests at the same time.

I've discovered multiple web requests sent by different threads are not
processed at the same time. They are processed practically sequentially. It
might be because there's a global internal lock in WebResponse or in
ConnectStream used by HttpWebResponse internally. Another possible reason
might be that CLR has a limit on the max amount of simultaneously running
threads, so it pauses my threads except few; but the first scenario looks
much more likely.

My logs show that about 2 web responses are processed at the same time:

Single thread request timing: 00:00:40.4820431
Starting 5 extra threads
Thread 2 started at 8/24/2006 3:23:08 PM
Thread 0 started at 8/24/2006 3:23:08 PM
Thread 3 started at 8/24/2006 3:23:08 PM
Thread 4 started at 8/24/2006 3:23:08 PM
Thread 1 started at 8/24/2006 3:23:08 PM
Thread 2 ended at 8/24/2006 3:24:01 PM, request timing: 00:00:52.8094580
Thread 0 ended at 8/24/2006 3:24:05 PM, request timing: 00:00:56.8873481
Thread 3 ended at 8/24/2006 3:24:43 PM, request timing: 00:00:42.0132049
Thread 4 ended at 8/24/2006 3:24:51 PM, request timing: 00:00:46.3567047
Thread 1 ended at 8/24/2006 3:25:16 PM, request timing: 00:00:33.4199499

I've also done tests with Socket class and made sure that sockets can
process multiple requests at the same time (unlike WebRequest / WebResponse).

I'd like to use sockets, but there's a problem with them, too: it does not
report the end of transmission correctly. My data receiving loop is simple:

while ((bytesCount = socket.Receive(buffer)) > 0)
{
    rawContentStringBuilder.Append(resultEncoding.GetString(buffer, 0,
bytesCount));
}

The problem is that Receive sometimes throws a timeout exception at the end
of transmission instead of returning 0 bytesCount. bytesCount also cannot
serve as a signal of transmission end because sometimes it is less then the
length of the buffer while there're more bytes to receive; as well, the size
of http content might be the same as the length of the buffer.
Socket.Available also doesn't signal the end of transmission - it seems to
return only the amount of bytes that have already come. So it can be 0 in the
middle of transition. Besides, if using sockets directly, I'd need to
implement processing of raw Http (headers, encodings, etc.).

So my questions are:

- is it possible to make (Http)WebResponse not being locked in threads
- how to determine the end of transmission with sockets

PS

1. I use .Net 1, although .Net 2 is also being considered for this
application.

2. The http requests are sent to the same url at the moment. Later they need
to be sent to different systems. The problem is not related to the server as
with sockets it works fine.